I managed to configure a shared dedicated hosting on a single IP with nginx. Default HTTP and HTTPS serving a 404 for unknown domains incoming.

1 - Create a default zone

As nginx is loading vhosts in ascii order, you should create a 00-default file/symbolic link into your /etc/nginx/sites-enabled.

2 - Fill the default zone

Fill your 00-default with default vhosts. Here is the zone i am using:

server {
    server_name _;
    listen       80  default_server;
    return       404;
}


server {
    listen 443 ssl;
    server_name _;
    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;
    return       404;
}

3 - Create self signed certif, test, and reload

You will need to create a self signed certificate into /etc/nginx/ssl/nginx.crt.

Create a default self signed certificate:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

Just a reminder:

• Test the nginx configuration before reloading/restarting : nginx -t
• Reload a enjoy: sudo service nginx reload

Hope it helps.

You do not have any ssl_certificate or ssl_certificate_key defined in your "default" https block. Although you do not have or want a real key for this default scenario, you still need to configure one or else nginx will have the undesired behaviour that you describe.

Create a self-signed certificate with a Common Name of * and plug it into your config and it will start to work as you want.

The "default" behaviour under this set up would be that a browser would get a warning that the certificate can not be trusted, if the user adds the certificate as an exception, the connection will be dropped by nginx and they will see their browser's default "could not connect" error message.

We basically want to avoid at all cost that the first server definition in our config file is served as a catch-all-server for SSL connections. We all know that it does that (opposed to http and using default_server config which works nicely).

This cannot be achieved declaratively for SSL (yet) so we have to code it with an IF...

The variable $host is the host name from the request line or the http header. The variable $server_name is the name of the server block we are in right now.

So if these two are not equal then you served this SSL server block for another host so that should be blocked.

The code does not contain specific references to your server IP addresses so it can easily be reused for other server configs without modification.

Example:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    ###
    ### Section: SSL

    #
    ## Check if this certificate is really served for this server_name
    ##   http://serverfault.com/questions/578648/properly-setting-up-a-default-nginx-server-for-https
    if ($host != $server_name) {
        #return 404 "this is an invalid request";
        return       444;
    }

    ...

To elaborate more on the Radmilla Mustafa's answer:

Nginx uses 'Host' header for server_name matching. It does not use TLS SNI. This means that for an SSL server, nginx must be able to accept SSL connection, which boils down to having certificate/key. The cert/key can be any, e.g. self-signed.

See documentation

Hence, the solution is:

server {
    server_name _;
    listen 80 default_server;
    listen 443 ssl default_server;

    ## To also support IPv6, uncomment this block
    # listen [::]:80 default_server;
    # listen [::]:443 ssl default_server;

    ssl_certificate <path to cert>;
    ssl_certificate_key <path to key>;
    return 404; # or whatever
}

If you just want to make sure no data is returned, you can use the following snippet without certificates:

map "" $empty {
        default "";
}

server {
        listen 80 default_server;
        listen 443 ssl http2 default_server;
        listen [::]:80 default_server;
        listen [::]:443 ssl http2 default_server;

        server_name _;

        ssl_ciphers aNULL;
        ssl_certificate data:$empty;
        ssl_certificate_key data:$empty;

        return 444;
}

To anyone who lost as much hair over this as me (spent nearly whole day on this today). I have tried almost everything, and the thing that made it finally working correctly was this stupid line:

ssl_session_tickets off;

Building upon Ifnot's answer, my working example is:

server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    server_name _;

    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;
    ssl_session_tickets off;

    return 404;
}

I have no idea why this was necessary, the only principle I derived out of this was that nginx behaves very strangely when we don't give him what he wants.

If you want to be absolutely sure, then use separate IP addresses for hosts which should not answer on HTTPS and hosts which should. This also resolves the "invalid certificate" browser warning problem.

TLS SNI

SNI allows browser to pass requested server name during the SSL handshake

Nginx support TLS SNI

To check if Nginx enabled TLS SNI

$ nginx -V
...
TLS SNI support enabled
...

and check the error_log that without this warning

nginx was built with SNI support, however, now it is linked
dynamically to an OpenSSL library which has no tlsext support,
therefore SNI is not available

Official HTTPS document has more detail.

If enabled TLS SNI, the following config works fine.

# Create the self signed certificate

openssl req -x509 -newkey rsa -nodes -keyout default.key -days 36500 -out default.crt -subj "/CN=example.org"

openssl req -x509 -newkey rsa -nodes -keyout a.key -days 36500 -out a.crt -subj "/CN=a.example.org"

openssl req -x509 -newkey rsa -nodes -keyout b.key -days 36500 -out b.crt -subj "/CN=b.example.org"

    server {
        listen       443 ssl default_server;
        server_name  "";

        ssl_certificate      default.crt;
        ssl_certificate_key  default.key;

        add_header "Content-Type" "text/plain";
        return 200 "default page";
    }

    server {
        listen       443 ssl;
        server_name  a.example.org;

        ssl_certificate      a.crt;
        ssl_certificate_key  a.key;

        add_header "Content-Type" "text/plain";
        return 200 "a.example.org page";
    }

    server {
        listen       443 ssl;
        server_name  b.example.org;

        ssl_certificate      b.crt;
        ssl_certificate_key  b.key;

        add_header "Content-Type" "text/plain";
        return 200 "b.example.org page";
    }

# Add -v to verify the certificate

$ curl --insecure --resolve "a.example.org:443:127.0.0.1" https://a.example.org

a.example.org page

$ curl --insecure --resolve "b.example.org:443:127.0.0.1" https://b.example.org

b.example.org page

$ curl --insecure https://127.0.0.1

default page

Ref: Nginx TLS SNI

If Nginx disable TLS SNI

Nginx will use default server certificate for all request

Official HTTPS document

This is caused by SSL protocol behaviour. The SSL connection is established before the browser sends an HTTP request and nginx does not know the name of the requested server. Therefore, it may only offer the default server’s certificate.