I am trying to configure two-way SSL with SSL certs (for server and client) signed by Intermediate CAs. This is what I have done so far following this tutorial.
Server - nginx application
Nginx is configured with SSL certificate (signed by an Intermediate CA).
server {
listen 443;
server_name app-ca.test.com;
ssl on;
ssl_certificate /root/ca/intermediate/certs/app-plus-intermediate.pem;
ssl_certificate_key /root/ca/intermediate/private/app-ca-interm-ca.test.com.key.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
# I have also tried adding the Intermediate CA cert in vain
# ssl_client_certificate /root/client_rootca_intermediate.crt;
ssl_client_certificate /root/client_rootca.crt;
ssl_verify_client on;
location / {
root /usr/share/nginx/massl;
index index.html index.htm;
}
}
Client - curl or OpenSSL s_client
I have a client certificate signed by some other Intermediate CA, which fails with 400 The SSL certificate error
I have also tried to pass (-cert
option in openssl
command) Client's Intermediate CA and Root CA along with the client certificate in vain.
$ cat /root/ca/intermediate/certs/client.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem > /root/ca/intermediate/certs/client_plus_intermediate.cert.pem
$ cat /root/ca/intermediate/certs/client.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem > /root/ca/intermediate/certs/intermediate_plus_client.cert.pem
$ cat /root/ca/intermediate/certs/client.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem /root/ca/certs/ca.cert.pem > /root/ca/intermediate/certs/client_plus_intermediate_plus_ca.cert.pem
Short Error Logs
<html>
<head><title>400 The SSL certificate error</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.13.5</center>
</body>
</html>
Long Error Logs
I have shortened the longs hashes for brevity.
$ openssl s_client -connect app-ca.test.com:443 -tls1 -key /root/ca/intermediate/private/client.key.pem -cert /root/ca/intermediate/certs/client.cert.pem -CAfile /root/server_rootca.crt -state -debug
CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x2239a90 [0x226e3c3] (181 bytes => 181 (0xB5))
0000 - 16 03 01 00 b0 01 00 00-ac 03 01 16 ed fa 81 3e ...............>
0010 - fc 25 c1 55 73 8a ca 5f-d3 56 11 a6 0f 38 6e 3c .%.Us.._.V...8n<
0020 - 52 fb 1f 9b fb 4f 4f 3e-5a fb 82 00 00 64 c0 14 R....OO>Z....d..
0090 - 00 ff 01 00 00 1f 00 0b-00 04 03 00 01 02 00 0a ................
00a0 - 00 0a 00 08 00 17 00 19-00 18 00 16 00 23 00 00 .............#..
00b0 - 00 0f 00 01 01 .....
SSL_connect:SSLv3 write client hello A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 42 ....B
read from 0x2239a90 [0x2269e78] (66 bytes => 66 (0x42))
0000 - 02 00 00 3e 03 01 6f e5-89 1d bd 5a 58 26 d7 11 ...>..o....ZX&..
0010 - 8a 05 fd 2a 04 96 58 2e-2e 19 a7 89 46 a0 5b 21 ...*..X.....F.[!
0020 - c3 90 1c 3e 0b e6 00 c0-14 00 00 16 ff 01 00 01 ...>............
0030 - 00 00 0b 00 04 03 00 01-02 00 23 00 00 00 0f 00 ..........#.....
0040 - 01 01 ..
SSL_connect:SSLv3 read server hello A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 0c ab .....
read from 0x2239a90 [0x2269e78] (3243 bytes => 3243 (0xCAB))
0000 - 0b 00 0c a7 00 0c a4 00-06 64 30 82 06 60 30 82 .........d0..`0.
0c90 - 5f b6 c7 86 5d 41 b3 fb-9c fe d3 0a 26 01 f9 d9 _...]A......&...
0ca0 - a6 ae 7f ff 4f c7 0b e8-97 b3 1c ....O......
depth=2 C = GB, ST = England, L = Melbourne, O = Alice Ltd, OU = IT Services, CN = server-and-ca.test.com, emailAddress = [email protected]
verify return:1
depth=1 C = GB, ST = England, O = Alice Ltd, OU = Shared Services, CN = server-and-interm-ca.test.com, emailAddress = [email protected]
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Alice Ltd, OU = Alice Ltd Web Services, CN = app-ca-interm-ca.test.com, emailAddress = [email protected]
verify return:1
SSL_connect:SSLv3 read server certificate A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 01 4b ....K
read from 0x2239a90 [0x2269e78] (331 bytes => 331 (0x14B))
0000 - 0c 00 01 47 03 00 17 41-04 13 5d 81 04 36 18 e7 ...G...A..]..6..
0010 - da bf 5e 30 dd d8 ee 77-f9 56 aa 77 8b 9e cd 3e ..^0...w.V.w...>.
0110 - d1 82 65 0f 5d 9c 03 ba-5f 7f 62 33 a8 a6 62 8e ..e.]..._.b3..b.
0120 - f2 5c 03 1d 4d 47 04 16-cb 80 09 39 32 be ca 23 .\..MG.....92..#
0130 - 41 95 36 a6 4b 6b f0 6c-df a5 4b 26 d4 4a c5 f3 A.6.Kk.l..K&.J..
0140 - 99 0d c8 d8 aa 5d f8 88-86 b3 15 .....].....
SSL_connect:SSLv3 read server key exchange A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 bc .....
read from 0x2239a90 [0x2269e78] (188 bytes => 188 (0xBC))
0000 - 0d 00 00 b4 03 01 02 40-00 ae 00 ac 30 81 a9 31 [email protected]
0010 - 0b 30 09 06 03 55 04 06-13 02 47 42 31 10 30 0e .0...U....GB1.0.
0090 - 06 09 2a 86 48 86 f7 0d-01 09 01 16 1b 72 6f 6f ..*.H........roo
00a0 - 74 40 63 6c 69 65 6e 74-2d 61 6e 64 2d 63 61 2e t@client-and-ca.
00b0 - 74 65 73 74 2e 63 6f 6d-0e test.com.
00bc - <SPACES/NULS>
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
write to 0x2239a90 [0x2273910] (1593 bytes => 1593 (0x639))
0000 - 16 03 01 06 34 0b 00 06-30 00 06 2d 00 06 2a 30 ....4...0..-..*0
0010 - 82 06 26 30 82 04 0e a0-03 02 01 02 02 02 10 00 ..&0............
05f0 - 29 2a 6c 40 d1 ed 8f 6d-15 b2 cd 6a 7b 72 30 91 )*[email protected]{r0.
0600 - ea 29 16 48 f2 11 21 15-3a 50 32 8b 95 87 b8 09 .).H..!.:P2.....
0610 - 11 84 9a a4 d2 b8 46 33-7a a2 79 51 ba 23 8c 96 ......F3z.yQ.#..
0620 - 45 62 2e b9 f5 ea 23 79-53 e0 cb 72 1f e6 19 d4 Eb....#yS..r....
0630 - 75 18 a8 2e 44 2f f3 8b-a7 u...D/...
SSL_connect:SSLv3 write client certificate A
write to 0x2239a90 [0x2273910] (75 bytes => 75 (0x4B))
0000 - 16 03 01 00 46 10 00 00-42 41 04 b9 b3 02 d2 bc ....F...BA......
0010 - e2 8b 49 a7 f6 8c 59 66-fc 0e 39 79 c7 23 34 e9 ..I...Yf..9y.#4.
0020 - 3e 04 98 3a 60 78 1d aa-51 06 46 80 09 10 c4 7e >..:`x..Q.F....~
0030 - a5 e7 05 d1 82 f2 0d bb-9a ca e7 29 01 0b 88 6d ...........)...m
0040 - ed c3 52 73 b1 d4 3a 95-00 e8 ..Rs..:...
004b - <SPACES/NULS>
SSL_connect:SSLv3 write client key exchange A
write to 0x2239a90 [0x2273910] (267 bytes => 267 (0x10B))
0000 - 16 03 01 01 06 0f 00 01-02 01 00 5e 29 8e 7c 69 ...........^).|i
0010 - 1e 10 0d 01 39 35 db 18-7e 4a a7 12 ae 12 7e f0 ....95..~J....~.
0020 - d6 93 c5 0a ba 5d e4 f1-a4 ae 8f c4 7d 52 80 16 .....]......}R..
00f0 - 6f 1f 56 73 bc ab 7f 07-1d f7 b4 ec d7 58 57 cd o.Vs.........XW.
0100 - cd e0 37 b3 58 09 3a 75-93 02 ab ..7.X.:u...
SSL_connect:SSLv3 write certificate verify A
write to 0x2239a90 [0x2273910] (6 bytes => 6 (0x6))
0000 - 14 03 01 00 01 01 ......
SSL_connect:SSLv3 write change cipher spec A
write to 0x2239a90 [0x2273910] (53 bytes => 53 (0x35))
0000 - 16 03 01 00 30 24 90 78-08 d3 10 f3 f8 e3 c8 86 ....0$.x........
0010 - 82 f1 54 d1 38 7b 57 7b-83 a3 49 b9 3b 80 b2 86 ..T.8{W{..I.;...
0020 - 54 74 92 ec 9a a7 e7 28-1a ec 72 4c 64 8e f3 e3 Tt.....(..rLd...
0030 - 08 96 89 2a 03 ...*.
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 06 ea .....
read from 0x2239a90 [0x2269e78] (1770 bytes => 1770 (0x6EA))
0000 - 04 00 06 e6 00 00 01 2c-06 e0 09 8d 58 07 45 c9 .......,....X.E.
0010 - 58 49 42 f4 13 00 47 12-be 22 a2 e3 a0 b6 22 bd XIB...G.."....".
06d0 - a1 11 26 db 43 c8 6e 47-2f 40 65 61 e1 4e ef 0a ..&.C.nG/@ea.N..
06e0 - 57 e0 28 19 2d 0d c6 7f-ae 2e W.(.-.....
SSL_connect:SSLv3 read server session ticket A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 14 03 01 00 01 .....
read from 0x2239a90 [0x2269e78] (1 bytes => 1 (0x1))
0000 - 01 .
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 30 ....0
read from 0x2239a90 [0x2269e78] (48 bytes => 48 (0x30))
0000 - 7d 5f 53 a4 5e 85 67 67-8d 6c d6 6e 93 cd c6 75 }_S.^.gg.l.n...u
0010 - c1 83 17 d9 a8 e3 89 23-86 6b 8a 04 2d 46 7e 95 .......#.k..-F~.
0020 - 15 46 a4 ec 73 f3 3d 78-1b 0e 94 62 79 cf 96 3d .F..s.=x...by..=
SSL_connect:SSLv3 read finished A
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Alice Ltd/OU=Alice Ltd Web Services/CN=app-ca-interm-ca.test.com/[email protected]
i:/C=GB/ST=England/O=Alice Ltd/OU=Shared Services/CN=server-and-interm-ca.test.com/[email protected]
1 s:/C=GB/ST=England/O=Alice Ltd/OU=Shared Services/CN=server-and-interm-ca.test.com/[email protected]
i:/C=GB/ST=England/L=Melbourne/O=Alice Ltd/OU=IT Services/CN=server-and-ca.test.com/[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGYDCCBEigAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgagxCzAJBgNVBAYTAkdC
MRAwDgYDVQQIDAdFbmdsYW5kMRIwEAYDVQQKDAlBbGljZSBMdGQxGDAWBgNVBAsM
zBcik+fj+MUtDzhEl6EuW1ILjAvt5u4KBxj6d0yAXzleACOYncYWWzMfQdrFmwKh
W2opZQ==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Alice Ltd/OU=Alice Ltd Web Services/CN=app-ca-interm-ca.test.com/[email protected]
issuer=/C=GB/ST=England/O=Alice Ltd/OU=Shared Services/CN=server-and-interm-ca.test.com/[email protected]
---
Acceptable client certificate CA names
/C=GB/ST=England/L=Sydney/O=Something/OU=Shared Services/CN=client-and-ca.test.com/[email protected]
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5682 bytes and written 2175 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID: 2AF7BFD60D3EC4686EAAAE1971FBD8999E65C5C80A32182CB9A668B1411DB09C
Session-ID-ctx:
Master-Key: B3F714B4ACB61C6310311025B25AFBAFA9E9AAEBB5ACD5FEEAE5DCAE2690DECBFA4EC5CBD2C8A50F349F43026CD0C564
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 09 8d 58 07 45 c9 58 49-42 f4 13 00 47 12 be 22 ..X.E.XIB...G.."
0010 - a2 e3 a0 b6 22 bd 0d 71-c9 46 bd ab 84 85 06 f7 ...."..q.F......
06b0 - 66 76 1f 3e 49 23 dc 2b-be 9e d5 03 b8 a5 a1 7d fv.>I#.+.......}
06c0 - 4d 56 79 3f 81 78 a1 11-26 db 43 c8 6e 47 2f 40 MVy?.x..&.C.nG/@
06d0 - 65 61 e1 4e ef 0a 57 e0-28 19 2d 0d c6 7f ae 2e ea.N..W.(.-.....
Start Time: 1506251677
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
GET / HTTP/1.0
write to 0x2239a90 [0x226e3c6] (90 bytes => 90 (0x5A))
0000 - 17 03 01 00 20 ca 44 95-8c a0 32 52 4d da d8 02 .... .D...2RM...
0010 - db bd 97 88 0e e3 cb b9-9e fb 50 7e 71 24 37 83 ..........P~q$7.
0020 - f8 48 03 a0 a1 17 03 01-00 30 db 99 b2 0c 6c e6 .H.......0....l.
0030 - f4 25 3d 54 2f b1 a3 3c-be 2a 36 94 6c ce 6d 8d .%=T/..<.*6.l.m.
0040 - 3d 54 82 d3 f0 2a 40 3d-fc 3f 1b 3e 4a 40 10 e5 =T...*@=.?.>J@..
0050 - 1d eb ab 00 69 f1 e0 4a-27 47 ....i..J'G
write to 0x2239a90 [0x226e3c6] (74 bytes => 74 (0x4A))
0000 - 17 03 01 00 20 95 06 3d-51 d5 7c c2 05 ef a7 d6 .... ..=Q.|.....
0010 - 2b 25 9c dd ec 5f 7c c0-15 83 c6 ca ea 47 a1 b2 +%..._|......G..
0020 - 82 2d 46 7d 64 17 03 01-00 20 3b 2e 36 63 10 b3 .-F}d.... ;.6c..
0030 - 50 c7 ec 36 a4 27 a0 4d-db bb 83 b5 c6 e8 d5 fa P..6.'.M........
0040 - ca 76 dc e7 63 8f 94 b3-24 3f .v..c...$?
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 17 03 01 01 a0 .....
read from 0x2239a90 [0x2269e78] (416 bytes => 416 (0x1A0))
0000 - a6 8b c1 bb a4 aa 12 2e-81 d9 45 41 74 0e 33 a4 ..........EAt.3.
0190 - 37 be 58 ca 01 80 fc 7c-79 2b 3f 54 a4 cd 4a 07 7.X....|y+?T..J.
HTTP/1.1 400 Bad Request
Server: nginx/1.13.5
Date: Sun, 24 Sep 2017 11:14:49 GMT
Content-Type: text/html
Content-Length: 231
Connection: close
<html>
<head><title>400 The SSL certificate error</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.13.5</center>
</body>
</html>
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 15 03 01 ...
0005 - <SPACES/NULS>
read from 0x2239a90 [0x2269e78] (32 bytes => 32 (0x20))
0000 - c3 75 ba 40 21 83 f7 0e-11 98 7b 44 84 bb 23 d5 .u.@!.....{D..#.
0010 - 80 32 1e 3e b6 b7 dd 4a-16 09 31 e9 62 a9 cd a3 .2.>...J..1.b...
SSL3 alert read:warning:close notify
closed
write to 0x2239a90 [0x226e3c3] (37 bytes => 37 (0x25))
0000 - 15 03 01 00 20 bd 18 f2-df 1b 84 fc 8e e0 80 a1 .... ...........
0010 - 2f 6f 31 b4 4c fc 1c e5-36 1f c5 fb 5d c0 f8 dc /o1.L...6...]...
0020 - 19 6b 03 c3 2d .k..-
SSL3 alert write:warning:close notify
Interestingly, the above command works fine if I use certificates of Client's Root CA or Client's Intermediate CA.
Finally, I have pinned down the root cause of the problem. There were two problems with my setup.
a) For two-way SSL, the certificate signed by the Intermediate CA must have
clientAuth
inextendedKeyUsage
(Thanks to @dave_thompson_085) which can be verified by the below commandb) Another, thing which was missing was ssl_verify_depth parameter in the nginx config file which must be
2
or more. It does not make much sense to make the number bigger than2
in my case, but it works with any number other than1
(which is default value). Interestingly, this is not required in nginx v1.12.X (my colleague with the exact same setup didn't have to specify this). However, it didn't work for me (nginx v1.13.5) until I used this parameter.I can have a sound sleep after 3 days of headbanging.
TIP: Don't depend on
curl
much to troubleshoot two-way SSL issues, tryopenssl s_client
instead.curl
can give misleading results sometimes, see this. I too fumbled around for a while in my Ubuntu 16.04 docker container.openssl s_client -cert $file
can provide only 'the' client cert (singular, one), not a chain; it can optionally provide the privatekey as well, if you don't specify-key
, but still not any cert(s) other than the client cert. If you put any additional cert(s) in that file they are totally and completely ignored.But libssl uses certs from the truststore to fill out the client's cert chain in addition to its nominal purpose of validating the server's cert chain. Read the subtly definitive words of the man page:
Since you need to specify your private root to verify the server chain, if you instead give
-CAfile
a file containing both your root and your intermediate then both directions will work.Unlike some other implementations where putting an intermediate in the truststore causes validation to stop at that point, OpenSSL will use truststore certs to complete a received chain that is incomplete, but by default still validates up to a root from the truststore; to stop earlier you must specify
-partial_chain
(new in 1.1.0).However, I'm surprised configuring the server with
ssl_client_certificate
containing root and intermediate didn't work. nginx also uses OpenSSL and as I said libssl will use intermediate(s) from truststore to complete (though not validate) a received chain. So that approach, although officially not standard-conforming (the sender is responsible for sending a full chain modulo root) should actually work. I'll try to test later when I have a chance.I had this issue when my computer (where I issue the cert) and the VM clock where I test my application where out of sync. And so
valid_from
in the VM didn't reach yet, and the check keep falling for me, until it suddenly start working (as the VM catch up eventually).found out by increasing the log verbosity on nginx to
debug
And i got this message in the log
and on the client side I was getting the
400
error message.Here is an example OpenSSL configuration file that generates the correct certificate signing request.
SSL certificates are dependable on a specific domain meaning if you have multiple ssl certs then these should be added to different server blocks. something like this.
also I notice in your sites nginx that only one cert is present unless this is a wildcard certificate you should have 2 certificates.
Hope this helps