How do I setup multiple subdomains with their own certificate using nginx?

Unless every answer that I've read was plain wrong, SNI should make it possible to do what I want, yet every guide tells me to do exactly what I'm doing.

And yet nginx is serving the wrong certificate so I'm clearly doing something wrong.

❯ sudo nginx -V | grep SNI                                                                                                                                                                                                                                                            %1
nginx version: nginx/1.10.3
built with OpenSSL 1.1.0f  25 May 2017
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-qJwWoo/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/ngi
nx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fa
stcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_reques
t_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --wit
h-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-qJwWoo/nginx-1.10.3/debian/modules/nginx-auth-pam --add-dynamic-module=/build/nginx-qJwWoo/nginx-1.10.3/debian/modules/nginx-dav-
ext-module --add-dynamic-module=/build/nginx-qJwWoo/nginx-1.10.3/debian/modules/nginx-echo --add-dynamic-module=/build/nginx-qJwWoo/nginx-1.10.3/debian/modules/nginx-upstream-fair --add-dynamic-module=/build/nginx-qJwWoo/nginx-1.10.3/debian/modules/ngx_http_substitutions_filter_m
odule

Here's what my configs look like:

server {
  listen 443 ssl default_server;
  listen [::]:443 ssl;

  server_name one.example.com;

  ssl on;
  ssl_certificate       /etc/letsencrypt/live/one.example.com/fullchain.pem;
  ssl_certificate_key   /etc/letsencrypt/live/one.example.com/privkey.pem;

  index index.html;
  root /var/www/one.example.com/site;
}

server {
  #listen 443 ssl default_server;
  listen [::]:443 ssl;

  server_name two.example.com;

  ssl on;
  ssl_certificate       /etc/letsencrypt/live/two.example.com/fullchain.pem;
  ssl_certificate_key   /etc/letsencrypt/live/two.example.com/privkey.pem;

  index index.html;
  root /var/www/two.example.com/site;
}

If I have the listen 443 ssl default_server; directive in either server it will return the SSL cert for that server for BOTH domains. If I remove it from both domains then I just get nothing at all - both server domains refuse connections.

What do I have going wrong here? Do I just not understand how SNI works? My nginx has been built with SNI support enabled. And yet... I only get the ssl cert served for one subdomain.