Home / server / Questions / 763567
Accepted
JayMcTee
Asked: 2016-03-15 02:22:32 +0800 CST

Nginx TLS Performance Optimization

  • 772

I switched a busy site to TLS-everywhere and HTTP/2 last week. Performance hasn't improved as much as hoped. In fact, depending on how you measure it, you can argue it has degraded.

There is plenty of scope to optimize the server further, at OS level TCP tweaks, TLS config in the web stack etc. To ensure I'm not overlooking any points for improvement, I'm posting my line of thought here for others to chime in.

Currently, the SSL conf shows up like this. Various parties control various elements of this, I don't have direct control over all of it. Domain name censored.

$ openssl s_client -state -CAfile Documents/Thawte\ Server\ CA.cer -connect xxxxxx.tld:443
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=3 /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=xxx
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF1----shZZU
y1kFXCCRo78=
-----END CERTIFICATE-----
subject=/C=xxx
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 6240 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES128-SHA
    Session-ID: 09464B1CF8972B08267C33BF255FDA74216728E3599B24EAE02B13642C268483
    Session-ID-ctx: 
    Master-Key: ACC3529AC7C80162797397D4E3041D1720EE4BA6A8BA5D2C200F8B48ADD6960CA9E48D5FF975472A9E8B43B7023CA4E4
    Key-Arg   : None
    Start Time: 1457708322
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
SSL3 alert read:warning:close notify
closed
SSL3 alert write:warning:close notify

Problems I see:

verify error:num=19:self signed certificate in certificate chain

I want the chain as short as possible, in the right order (it isn't according to ssllabs.com) and there shouldn't be any need for self-signed certificates in the chain. Edit: I see this is actually the CA root certificate, which is already in the browser's trust store. This should be removed anyhow.

SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server key exchange A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A

This handshake sequence seems long to me. I've seen fewer back-and-forths on other configurations. My desire is to keep round trips to a minimum.

Certificate Chain 0,1,2,3 - made up of 4 (Comodo) certificates, it is over 5.5Kb and hence may not fit in the TCP congestion window size (not verified yet). Probably the cause for extra round trips (?).

Perhaps it's worth switching to a CA which offers shorter chains. Any suggestions?

Cipher : DHE-RSA-AES128-SHA

Weak Diffie-Hellman cipher. SSLLabs adds: "Uses common DH primes Yes Replace with custom DH parameters if possible"

My questions hence are:

  • Which ways can I shorten the certificate chain?
  • Which ways can I reduce the handshake download size?
  • Which ways can I reduce the handshake round trips?
  • Which CA offers shorter chains than Comodo?
  • Do you see any other problem points in this output? Or what else should I test?
  • Which ways can we optimize TCP config on Ubuntu to maximize TLS performance / network throughput?
  • What is the best way to pick cipher suite order for performance? I ran $ openssl speed but not sure how to interpret and action this data.

Your insights would be much appreciated.

nginx

1 Answers

  1. Best Answer

    Lets sort the certificate chain issues first. This should take care of the first 4 points.

    You will have been sent a certificate from Comodo and you will also need some more of the certificate chain. A list of the certificates can be found here. You will need to inspect the certificate to find out what CA it was signed with.

    Once you have identified the CA download the bundle and place it in the same directory as the certificate you were sent. You will then need to combine them using the following command

    cat name_of_file_you_were_sent name_of_bundle_file > www.example.com.crt

    You may notice that the final CA is "missing" from the bundle, this won't stop you passing this part of the test on ssllabs.

    You will then need to reference this file along with the private key file in the nginx config (Which you obviously have done but for completeness) using these two config lines

    ssl_certificate     www.example.com.crt;
ssl_certificate_key www.example.com.key;

    Test nginx config and restart if everything is ok

    As for the rest of the configuration.

    The best site I have found for generating an SSL config for nginx is Mozilla Security I've not included an example as there are many on serverfault alone

    Depending on your site traffic you will either want to use intermediate or Modern (Hopefully you have no need to use old and if you do you may want to deal with that, but that's a different question)

    You shouldn't just copy and paste those configs without understanding what the settings will do. HSTS for example can have long term impact on site visitors.

    While in general these configuration are "safe" they can be optimised a little more. Specifically

    ssl_buffer_size 16k; #Better if you have video
ssl_buffer_size 4k; #Better for quicker first byte.

    Of course there could be some other reasons for why the switch to http/2 did not increase your site performance.

    For example http/2 opens many more connections to the server. Did you account for this and adjust worker_processes and worker_connections but the above deal with anything directly related to the SSL config parts of nginx.

    • 4