Greg Dubicki's questions

I use nginx 1.10.1 with config similar to this:

server {

    (...)

    add_header Header1 "value";

    (...)

    # in this location above add_header directive works
    location / {
         uwsgi_pass unix:/var/run/some.sock;

         (...)
    }

    # ..and it this one it doesn't!
    location ~* (^/$|favicon.ico|robots.txt) {
        return 204;
        expires 24h;
        add_header Cache-Control "public";
        etag on;
    }
}

..so my problem is that Header1 is set for requests processed by the 1st location, but not for the 2nd one.

Why?

I have read add_header docs and know that it works by default only for "positive" return codes, but 204 is one of them (I have actually tested changing the code to 200, 404 and it didn't help).

(I have also tried to add always to my add_header Header1 ... but it was a rather desperate try as it shouldn't help - and it didn't.)

I am trying to get the exit code of the last command chained with && and || correctly.

I encountered a strange behavior that I cannot explain. Please help.

This is my exit-code.sh test script:

#!/bin/bash
echo "running exit-code with $1"
exit $1

This works as expected:

$ ./exit-code.sh 1 && ./exit-code.sh 2 && ./exit-code.sh 3 || echo last exit code: $?
running exit-code with 1
last command exit code: 1

The same with () to run it in a subshell:

$ (./exit-code.sh 1 && ./exit-code.sh 2 && ./exit-code.sh 3 || echo last exit code: $?)
running exit-code with 1
last exit code: 1

But:

$ /bin/bash -c "./exit-code.sh 1 && ./exit-code.sh 2 && ./exit-code.sh 3 || echo last exit code: $?"
running exit-code with 1
last exit code: 0

Why do I get exit code 0 here?

I have a:

• single instance of Apache Kafka (2.9.2-0.8.1.1) installed from the source, using the default configuration (server.properties etc.),
• Zookeeper (3.4.5+dfsg-2) stand-alone, installed from the package

..running on a Ubuntu 14.04 LTS host with a name hostname1.

I try to shutdown Kafka cleanly but I can't.

I tried 2 methods:

• sending SIGINT - but Kafka simply ignores it (doesn't mention recieving the signal in any log, keeps running) [SIGTERM works but that's not the point]

• using Controlled Shutdown tool - but it fails with the following..

..stacktrace:

[2015-02-03 12:55:49,461] ERROR Operation failed due to controller failure (kafka.admin.ShutdownBroker$)
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: hostname1; nested exception is:
        java.net.ConnectException: Connection refused]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
        at kafka.admin.ShutdownBroker$.kafka$admin$ShutdownBroker$$invokeShutdown(ShutdownBroker.scala:56)
        at kafka.admin.ShutdownBroker$.main(ShutdownBroker.scala:109)
        at kafka.admin.ShutdownBroker.main(ShutdownBroker.scala)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: hostname1; nested exception is:
        java.net.ConnectException: Connection refused]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
        at javax.naming.InitialContext.lookup(InitialContext.java:411)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:286)
        ... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: hostname1; nested exception is:
        java.net.ConnectException: Connection refused
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
        ... 9 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 14 more 

I have SSH root access to an EC2 instance.

It's Ubuntu 14.04 LTS.

I have nginx web server listening on all interfaces on port 80 TCP. It's accessible from this server but not from the outside.

Apparently all of my incoming traffic except 22 TCP is blocked - other TCP ports, all UDP and ICMP.

Yet my iptables on this server are empty:

root@ip-x-x-x-x:~# iptables -v -L
Chain INPUT (policy ACCEPT 3162 packets, 1472K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 4203 packets, 674K bytes)
 pkts bytes target     prot opt in     out     source               destination  

Here's the nmap scan output from my local machine:

grzegorz@MacBook-Pro-Grzegorz:~$ sudo nmap -v -Pn -p 22,80 x.x.x.x

Starting Nmap 6.47 ( http://nmap.org ) at 2015-01-31 20:40 CET
Initiating Parallel DNS resolution of 1 host. at 20:40
Completed Parallel DNS resolution of 1 host. at 20:40, 0.16s elapsed
Initiating SYN Stealth Scan at 20:40
Scanning ec2-x-x-x-x.eu-central-1.compute.amazonaws.com (x.x.x.x) [2 ports]
Discovered open port 22/tcp on 54.93.91.112
Completed SYN Stealth Scan at 20:40, 1.38s elapsed (2 total ports)
Nmap scan report for ec2-x-x-x-x.eu-central-1.compute.amazonaws.com (x.x.x.x)
Host is up (0.035s latency).
PORT   STATE    SERVICE
22/tcp open     ssh
80/tcp filtered http

Read data files from: /usr/local/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 1.58 seconds
           Raw packets sent: 3 (132B) | Rcvd: 1 (44B)

..so some kind of firewall is blocking my traffic.

Let's assume that it's NOT an EC2 Security Group.

What else can be blocking my traffic? Something that is running on the VPS itself? Something in the kernel or something in the user space?

UPDATE:

I tested apparmor:

root@ip-x-x-x-x:~# apparmor_status --verbose
apparmor module is loaded.
4 profiles are loaded.
4 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/tcpdump
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
   /sbin/dhclient (607) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

..and I interpret this output as it hadn't anything to do with the filtering but I have disabled it anyway:

root@ip-x-x-x-x:~# service apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
root@ip-x-x-x-x:~# service apparmor teardown
 * Unloading AppArmor profiles
   ...done.

..but it didn't help. nmap still shows filtered TCP 80, nginx still not accessible.

I have SSH root access to an EC2 instance but no access to the AWS Console / AWS KEY & SECRET.

My incoming traffic to this host on all TCP ports except 22 TCP seems to be blocked - I cannot access my services (for example nginx on port 80) from the outside.

nmap shows these ports as filtered, while port 22 is shown as open when SSH is running and closed when SSH is temporarily shut down.

ICMP and UDP are also blocked.

(I used ping, nc and some other tools to check that.)

I know that my instance is in some custom, non-default EC2 Security Group named, let's say my-security-group, but I don't know its rules.

How to list these rules with the access level I have got?

Update 1: My iptables rules are empty - let's assume that I am sure that its the Security Group that is blocking my traffic.

I am using Ubuntu 12.04 LTS and I have apparently messed up my system with various versions of duplicity installations:

• first I had 0.6.18 installed from the default repo,
• then I tried using 0.6.23 from the PPA
• in the end I built 0.6.24 from the source and installed it

(I list this for completeness but I don't really think this is duplicity-specific.)

Now my system behaves like this:

root@host:~# duplicity -V
duplicity 0.6.24

root@host:~# which -a duplicity
/usr/bin/duplicity

root@host:~# /usr/bin/duplicity -V
duplicity 0.6.24

root@host:~# head /usr/bin/duplicity
#!/usr/bin/python2.7
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# duplicity -- Encrypted bandwidth efficient backup
# Version 0.6.23 released $reldate
#
# Copyright 2002 Ben Escoto <[email protected]>
# Copyright 2007 Kenneth Loafman <[email protected]>
#
# This file is part of duplicity.

???

Is there some kind of binary cache in Ubuntu that is being used here?

UPDATE:

root@host:~$ cat /usr/bin/duplicity | grep 24
            return 128 * 1024
    if maxopen < 1024:
        log.FatalError(_("Max open files of %s is too low, should be >= 1024.\n"
                         "Use 'ulimit -n 1024' or higher to correct.\n") % (maxopen,),
root@host:~$ cat /usr/bin/duplicity | grep 23
# Version 0.6.23 released $reldate
log.Log(u"duplicity 0.6.23 ($reldate)", verbosity)

Before

Until recently I used nginx 1.4.x with zebrafishlabs/nginx-statsd as a reverse proxy for openresty (yes, I know, nginx behind nginx is stupid - keep reading please) with the following location nginx configuration:

location /track {
  statsd_count "tracker.track_$scheme" 1;
  proxy_pass http://localhost:10081/track;
  proxy_set_header Host $host;
}

..and the following location openresty (listening on port 10081 TCP, of course) config:

location /track {
  content_by_lua_file 'conf/track.lua';
}

..and it worked. Please note that I have a few locations like that - it is important, you will see later on.

Stats were being pushed to statsd and the scripts were executed.

Now

Recently I decided to remove the front end nginx and use openresty alone.

Now I am using openresty 1.5.12.1 with zebrafishlabs/nginx-statsd compiled in and the following (obvious) configuration:

location /track {
  statsd_count "tracker.track_$scheme" 1;
  content_by_lua_file 'conf/track.lua';
}

But after this change sometimes my stats are not being pushed to statsd.

Apparently it happens if my lua script end execution with ngx.redirect lua command..

I tried to use this config as a workaround:

location /track {
  statsd_count "tracker.track_$scheme" 1;
  proxy_pass http://127.0.0.1/track-do;
}

location /track-do {
  content_by_lua_file 'conf/track.lua';
}

...but I got really strange results:

When I changed my "track" location this way it started to work properly - all stats pushed to statsd, code executed.

..but at the same time my other location, "static", that was not changed stopped pushing anything to statsd (!).

I think it is related to nginx processing phases. zebrafishlabs/nginx-statsd does do logging in the last, log phase while content_by_lua_file is processed in the content phase and ngx.redirect's that my scripts use documentation states that it "(...) terminates the current request's processing and never returns." but proxy_pass is also in the content phase and statsd directives were run when I used them together in nginx...

1. Can you help me in this specific problem?

2. Do you have any success using zebrafishlabs/nginx-statsd with openresty? How do you do that?

PS I already thought of using lonelyplanet/openresty-statsd and it would probably be a working workaround but I would have to move all my statsd usage inside my script which I do not want to keep them clean.

UPDATE: lonelyplanet/openresty-statsd way also didn't work!?