I've blindly followed some podman tutorial (German), which tricked me into creating a Kubernetics container.

I guess these commands caused that:

$ podman pod create --name somename
$ podman run --detach --pod=somename alpine:latest top

So I accidentally created that and even after podman system prune -a or so, the image k8s.gcr.io/pause and a container that calls itself an „infra container” and refuses to be removed remains.

Here some commands:

$ podman images                  
REPOSITORY                TAG     IMAGE ID      CREATED         SIZE
[…]
k8s.gcr.io/pause          3.2     80d28bedfe5d  5 months ago    688 kB
$ podman ps -a
CONTAINER ID  IMAGE                 COMMAND  CREATED       STATUS   PORTS   NAMES
a19af97ae7ca  k8s.gcr.io/pause:3.2           26 hours ago  Created          6e40c6882a16-infra
$ podman rm 6e40c6882a16-infra 
Error: container a19af97ae7caa0372394af71c9b10a66dd373d699d8a60d81ff7bf8c1749d3d4 is the infra container of pod 6e40c6882a16f884ee953beb4c36a548590a06006c502e00653e807c474ffaa6 and cannot be removed without removing the pod
$ podman rmi k8s.gcr.io/pause:3.2 
Error: 1 error occurred:
    * could not remove image 80d28bedfe5dec59da9ebf8e6260224ac9008ab5c11dbbe16ee3ba3e4439ac2c as it is being used by 1 containers: image is being used

Note I do not use kubectl – nor do I have this installed. So this answer for Kubernetics did not help me.

Similar to this and this question I want to block users from using the IP to access my server.

For HTTP (port 80) this works fine, but not for HTTPS. So users can still enter https://<myip> to access the webserver and nginx returns the default certificate. BTW I use HTTP2 in my "usual" server blocks (with my domain names), so I use:

listen 443 ssl http2;
listen [::]:443 ssl http2;

I tried this to block HTTPS ip access now:

server {
   listen 443 ssl;
   listen [::]:443 ssl;

   server_name _;
   return 444;
}

However unfortunately this blocks all HTTPS requests no matter whether the domain is used or not. I know it might be necessary to block non-SNI clients as these of course do not deliver the used domain name to the server, so I am fine with that. (I mean clients not supporting SNI are old anyway...)

I generally prefer to block this in nginx, but if you would have some ideas about blocking this at the firewall level (iptables) I would also happily appreciate these too. And if you want to argue why blocking with iptables is better you can also do this and convince me to also block HTTP [or all other] requests to the IP too. Generally dropping the connection is fine (as nginx status code 444 does).

However there is one requirement: I do not want to explicitly mention the IP address of the server in the config as it is a dynamic IP and the server uses a dynamic dns service.

So in short, here is what I want to achieve:

• block access via IP
• allow access via domain name
• it is fine to block non-SNI clients
• it is fine to use a firewall blocking for this
• dropping the connection is fine
• without mentioning the IP address of the server
• no exposing of the domain name via HTTPS

Edit: Another failed try. I tried to follow this suggestion and use this config snippet, which seems logically as a good thing:

if ($host != "example.com") {
        return 444;
}

It also basically works, but when I access "https://" I see that nginx at first already sends the HTTPS certificate (containing the domain name) and only when I skip the connection warning, I see that it blocks access. This is logical as nginx can only read the Host header when the HTTPS connection is there, but at this point nginx already send the server certificate containing the domain name, so a user now has the domain name and can reconnect using it making the whole IP blocking useless. I am also a bit concerned about the performance aspect of this solution as it causes nginx to check the host header for every request, so I am still looking for a solution here.

I want to use the new cookie-prefixes, which are not yet standardized by the IETF. These are __Secure- and __Host-.

So let's e.g. set this cookie (here the header returned by the server):

Set-Cookie: "__Host-apple=yummy; Secure; HttpOnly; Path=/"

I want to access this cookie now in nginx with the $cookie- variable. So for testing I use the echo module to show me the value of the cookie:

location = /wannaeat/ {
        echo $cookie___Host-apple;
}

However nginx always shows me -apple. It seems to use the hyphen (-) to split the variable.

Because when I e.g. name the cookie __Host_apple (and the nginx variable $cookie___Host_apple) nginx shows me the value correctly.

I know I could probably use $http_cookie and use a regular expression to find the correct cookie, but this is not possible as I want to map the cookie value.

Especially when configuring HPKP header (or other long headers in general) it would be useful to split a line in an nginx config over multiple lines.

This is the desired result:

pin-sha256="X3pGTSOuJeEVw989IJ/cEtXUEmy52zs1TZQrU06KUKg=";
pin-sha256="MHJYVThihUrJcxW6wcqyOISTXIsInsdj3xK8QrZbHec="; 
pin-sha256="isi41AizREkLvvft0IRW4u3XMFR2Yg7bvrF7padyCJg="; 
pin-sha256="I/bAACUzdYEFNw2ZKRaypOyYvvOtqBzg21g9a5WVClg="; 
pin-sha256="Y4/Gxyck5JLLnC/zWHtSHfNljuMbOJi6dRQuRJTgYdo="; 
pin-sha256="/oCVQg3nP3DroGpFdAbaiYzenycUftqrH3LAyaIal2g=";

However for the browser it should only be one line:

pin-sha256="X3pGTSOuJeEVw989IJ/cEtXUEmy52zs1TZQrU06KUKg="; pin-sha256="MHJYVThihUrJcxW6wcqyOISTXIsInsdj3xK8QrZbHec="; pin-sha256="isi41AizREkLvvft0IRW4u3XMFR2Yg7bvrF7padyCJg="; pin-sha256="I/bAACUzdYEFNw2ZKRaypOyYvvOtqBzg21g9a5WVClg="; pin-sha256="Y4/Gxyck5JLLnC/zWHtSHfNljuMbOJi6dRQuRJTgYdo="; pin-sha256="/oCVQg3nP3DroGpFdAbaiYzenycUftqrH3LAyaIal2g=";

So I've tried some things, but I'm not satisfied with the results...

First try: Just split it

add_header Public-Key-Pins '
pin-sha256="X3pGTSOuJeEVw989IJ/cEtXUEmy52zs1TZQrU06KUKg=";
pin-sha256="MHJYVThihUrJcxW6wcqyOISTXIsInsdj3xK8QrZbHec=";
pin-sha256="isi41AizREkLvvft0IRW4u3XMFR2Yg7bvrF7padyCJg=";
pin-sha256="I/bAACUzdYEFNw2ZKRaypOyYvvOtqBzg21g9a5WVClg=";
pin-sha256="Y4/Gxyck5JLLnC/zWHtSHfNljuMbOJi6dRQuRJTgYdo=";
pin-sha256="/oCVQg3nP3DroGpFdAbaiYzenycUftqrH3LAyaIal2g=";
'

This works, but with curl I can see that the browsers receives the header with all line breaks...

Second try: Backslash

Actually in the already linked article Scott Helme recommends this:

add_header Public-Key-Pins ' \
pin-sha256="X3pGTSOuJeEVw989IJ/cEtXUEmy52zs1TZQrU06KUKg="; \
pin-sha256="MHJYVThihUrJcxW6wcqyOISTXIsInsdj3xK8QrZbHec="; \
pin-sha256="isi41AizREkLvvft0IRW4u3XMFR2Yg7bvrF7padyCJg="; \
pin-sha256="I/bAACUzdYEFNw2ZKRaypOyYvvOtqBzg21g9a5WVClg="; \
pin-sha256="Y4/Gxyck5JLLnC/zWHtSHfNljuMbOJi6dRQuRJTgYdo="; \
pin-sha256="/oCVQg3nP3DroGpFdAbaiYzenycUftqrH3LAyaIal2g="; \
'

However in my case this just added the slashes and also returned them to the browser, so this does not work.

So how can I do this?

Bonus

Of course comments for every line would be an awesome addition:

pin-sha256="X3pGTSOuJeEVw989IJ/cEtXUEmy52zs1TZQrU06KUKg="; # current ECDSA
pin-sha256="MHJYVThihUrJcxW6wcqyOISTXIsInsdj3xK8QrZbHec="; # current RSA (nginx 1.11.0)
pin-sha256="isi41AizREkLvvft0IRW4u3XMFR2Yg7bvrF7padyCJg="; # backup ECDSA 1
pin-sha256="I/bAACUzdYEFNw2ZKRaypOyYvvOtqBzg21g9a5WVClg="; # backup ECDSA 2
pin-sha256="Y4/Gxyck5JLLnC/zWHtSHfNljuMbOJi6dRQuRJTgYdo="; # backup RSA 1
pin-sha256="/oCVQg3nP3DroGpFdAbaiYzenycUftqrH3LAyaIal2g="; # backup RSA 2

The situation

I am running Etherpad, which is proxied via nginx. Etherpad uses Websockets with Socket.io.

My nginx config is more or less this one. The location block for socket.io is this:

rewrite /CustomSubDir/socket.io/(.*) /socket.io/$1 break;
proxy_pass http://localhost:CustomPort/;
proxy_redirect   / /CustomSubDir/;
proxy_cookie_path / /CustomSubDir/;

# usual proxy header
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;

# websocket
proxy_set_header Accept-Encoding "";
proxy_http_version 1.1;
# http://nginx.org/en/docs/http/websocket.html
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;

proxy_buffers 8 32k;
proxy_buffer_size 64k;

proxy_buffering off;

What's confusing

So the good message is: Everything is working! I am therefore not asking how to get anything working.
The bas one is: Although everything works nginx always shows me error messages.

What's wrong

nginx shows me these errors in the error.log:

2016/05/24 xx:yy:zz [error] 22197#0: *12059 connect() failed (111: Connection refused) while connecting to upstream, client: SOM.IPA.DDR.EES, server: somedomain.example.com, request: "GET /CustomSubDir/socket.io/?EIO=3&transport=polling&t=1464121868147-3&sid=H2GhIY24t2a40etpAACd HTTP/2.0", upstream: "http://[::1]:CustomPort/socket.io/?EIO=3&transport=polling&t=1464121868147-3&sid=H2GhIY24t2a40etpAACd", host: "somedomain.example.com"
2016/05/24 xx:yy:zz [error] 22197#0: *12070 connect() failed (111: Connection refused) while connecting to upstream, client: SOM.IPA.DDR.EES, server: somedomain.example.com, request: "POST /CustomSubDir/socket.io/?EIO=3&transport=polling&t=1464122037998-5&sid=T-pthraR669TF2cKAACe HTTP/2.0", upstream: "http://[::1]:CustomPort/socket.io/?EIO=3&transport=polling&t=1464122037998-5&sid=T-pthraR669TF2cKAACe", host: "somedomain.example.com"

So I could trace this requests down. Here is why: 1. These are obviously Websocket requests. 2. And these are - and that's special - POST requests.

When loading an Etherpad or reconnecting to one after a lost connection only one request is made, which fulfils these requirements. I can clearly se it in the browser and can see it appearing in the nginx error log in real-time. This is the request in my browser:

200 POST https://somedomain.example.com/CustomSubDir/socket.io/?EIO=3&transport=polling&t=1464121868143-2&sid=H2GhIY24t2a40etpAACd

And it contains (e.g.) this payload:

164:42["message",{"component":"pad","type":"CLIENT_READY","padId":"CENSORED","sessionID":"null","password":null,"token":"t.qbszmj[...]","protocolVersion":2}]

The server reply is:

HTTP/2.0 200 OK
Server: nginx
Date: Tue, 24 May 2016 xx:yy:zz GMT
Content-Type: text/html
Content-Length: 2
access-control-allow-origin: *
Set-Cookie: io=H2GhIY24t[...]
X-Firefox-Spdy: h2

ok

Why I can be sure this is the culprit

I am very sure the POST request causes this. Not only because it is the only POST request with this URL when accessing the pad, I can also compare the behaviour. Because on the same server I also run Etherdraw, which works very similar, but it has one important difference: It does not seem to use POST Websocket requests.
And guess what? The error.log is empty.

My questions

So what are my questions:

1. How I can see the request to be successful in my browser (with a correct server reply!) while nginx reports me it is failing in the log?
2. How can I get rid of these error messages? AFAIK the requests do not fail...

I have self-compiled nginx 1.9.14 with http2 support:

nginx version: nginx/1.9.14
built by gcc 4.9.2 (Debian 4.9.2-10) 
built with OpenSSL 1.0.1k 8 Jan 2015
TLS SNI support enabled

So I have a location = / block in my config, which should add a header only when the user accesses the root site (https://myserver.example.com/). However when I access my server in Firefox the header in the / location block (which is WhereIAm) is not added:

Content-Encoding: gzip
Content-Type: text/html
Date: Thu, 21 Apr 2016 07:12:11 GMT
Etag: W/"570bf892-363"
Last-Modified: Mon, 11 Apr 2016 19:18:42 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000; includeSubDomains;  preload
X-Content-Type-Options: nosniff
X-Firefox-Spdy: h2
headerend: AtTheEnd
testing: IsTLS
x-something: ALsoAtTheEnd

This is my nginx config:

server {
        listen 80;
        listen [::]:80;

        server_name myserver.example.com;
        root /var/www/html;

        server_tokens off;

        add_header testing IsHTTP;
}

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        server_name myserver.example.com;
        root /var/www/html;

        server_tokens off;

        index index.php index.html index.htm;

        # -- TEST BEGIN --

        location = /configtest {
                #echo "itworks";
                add_header itworks configtest;
        }

        add_header testing IsTLS;

        location = / {
                add_header WhereIAm Is:/;
        }
        location /testdir1 {
                add_header WhereIAm Is:/testdir1;
        }
        location = /testdir2 {
                add_header WhereIAm Is:/testdir2;
        }

        # -- TEST END --


        # -- PARSER --

        location ~ \.php$ {
                #location ~ \..*/.*\.php$ {return 404;}
                include snippets/fastcgi-php.conf;
        #
        #       # With php5-cgi alone:
        #       fastcgi_pass 127.0.0.1:9000;
        #       # With php5-fpm:
                fastcgi_pass unix:/var/run/php5-fpm.sock;
        }

        # -- HTTPS --

        # certs sent to the client in SERVER HELLO are concatenated in ssl_certificate
        ssl_certificate /path/to/my/certificate;
        ssl_certificate_key /path/to/my/privatekey;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security "max-age=15768000; includeSubDomains;  preload";

        # OCSP Stapling ---
        # fetch OCSP records from URL in ssl_certificate and cache them
        ssl_stapling on;
        ssl_stapling_verify on;

        ## verify chain of trust of OCSP response using Root CA and Intermediate certs
        ssl_trusted_certificate /path/to/the/rootandintermediatecerts;

        # more headers
        add_header X-Something ALsoAtTheEnd;
        add_header HeaderEnd AtTheEnd;
        add_header X-Content-Type-Options nosniff;

        # deny access to Apache config files
        location ~ /\.ht {
                deny all;
        }
}

So in contrast when I access https://myserver.example.com/testdir1/ (and testdir1 is a directory, which exists on my server) it adds the correct header:

Content-Encoding: gzip
Content-Type: text/html
Date: Thu, 21 Apr 2016 07:24:22 GMT
Etag: W/"571538f2-363"
Last-Modified: Mon, 18 Apr 2016 19:43:46 GMT
Server: nginx
X-Firefox-Spdy: h2
whereiam: Is:/testdir1

What's more confusing is when I change the location block to this:

location = / {
    echo "ThisIsATest";
    add_header WhereIAm Is:/;
}

(I've compiled nginx with the echo-nginx-module)

Nginx does not only return the test string when accessing it, but it also adds the correct header:

Content-Type: application/octet-stream
Date: Thu, 21 Apr 2016 07:32:01 GMT
Server: nginx
X-Firefox-Spdy: h2
whereiam: Is:/

So what is going on here and how can I get nginx to only add a header when a user accesses /?