Parker's questions -server

Parker

Asked: 2019-08-20 06:05:20 +0800 CST

How to verify Certificate Revocation List(s) against multiple certification paths

2

In a recent question, I outlined the steps for verifying a wildcard SSL certificate for connecting to PostgreSQL from a remote client (using the same wildcard certificate I use for my web server). Although I resolved that problem, one lingering thing I haven't yet figured out is how to confirm I have the correct CRL(s) for my certificate.

My certificate has two Certification Paths (screenshot from SSLLabs test):

However, the CA only included the CRL for one of these paths (Path #2) with their issued certificate. I believe I have identified the correct CRL for the second path, but I'm not sure how to check it against my SSL certificate to confirm that it is correct. I can visually confirm that the CN matches, but there must be some OpenSSL command to validate the CRL against the corresponding certificate. I've come to the conclusion that I can't use the CRL fingerprints for this, since they change with each version of the CRL.

Question:

How can I validate each individual CRL against its specific trusted path via OpenSSL (either separately or in bundles)?

Can these two CRLs be bundled in the same way that certificates can be bundled (i.e., by simply appending PEM files)?

Essentially, I'm looking for a definitive way to test that "yes, this CRL (bundle?) contains the correct CRL for this certificate". With certificates and private keys, I can test the modulo, but would that be strong enough for a root certificate/CRL?

Here are the specific root certificates and CRLs I am working with (same as in above image).

Path #1 Root

Root certificate for Path #1 (USERTrust RSA Certification Authority):

# openssl x509 -fingerprint -noout -in USERTrustRSACertificationAuthority.crt
SHA1 Fingerprint=2B:8F:1B:57:33:0D:BB:A2:D0:7A:6C:51:F7:0E:E9:0D:DA:B9:AD:8E

This is what I presume to be the correct CRL for above root certificate:

Download CRL from CA:

# wget http://crl.usertrust.com/USERTrustRSACertificationAuthority.crl

Convert DER to PEM:

# openssl crl -inform DER -in USERTrustRSACertificationAuthority.crl -outform PEM -out USERTrustRSACertificationAuthority.pem

Get the fingerprint of the CRL:

# openssl crl -fingerprint -noout -in USERTrustRSACertificationAuthority.pem
SHA1 Fingerprint=BA:32:A2:48:F9:72:C9:01:44:17:B0:EE:6D:7F:AB:29:50:6F:A2:D4

Print the text version of the CRL:

# openssl crl -in USERTrustRSACertificationAuthority.pem -text
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha384WithRSAEncryption
        Issuer: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
        Last Update: Aug 18 13:58:25 2019 GMT
        Next Update: Aug 22 13:58:25 2019 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB

            X509v3 CRL Number:
                3628
Revoked Certificates:
    Serial Number: D8C857CAE4BE66F913759FDDA3D5DB2B
        Revocation Date: Mar 16 12:21:54 2018 GMT
    Serial Number: A6DC2ED38830CC033D23ABCEB72564BF
        Revocation Date: Mar 16 12:21:54 2018 GMT
    Serial Number: 3291B976375B708A08E5322427BBC434
        Revocation Date: Mar 16 12:21:54 2018 GMT
    Signature Algorithm: sha384WithRSAEncryption
         65:73:d1:c2:b2:21:83:6b:10:9c:0d:83:e7:07:32:7a:8b:25:
         bf:1c:84:03:7b:f5:8f:c1:3b:4e:3f:b3:0c:c9:61:1d:be:4c:
         92:5e:cd:fe:56:07:0b:ff:52:38:f1:1b:0e:ad:40:a7:d8:28:
         59:76:c2:e7:24:77:29:71:e5:53:4c:2f:c3:65:ce:b3:49:9c:
         45:d3:4a:0a:a5:75:00:1f:65:85:9c:16:e2:d7:38:6f:ea:bf:
         d3:58:ef:7e:cb:10:7a:c8:9b:bb:e4:69:e6:35:b4:31:88:19:
         29:dc:2b:87:95:4f:9e:17:0a:f5:16:b7:70:1f:2a:7c:d0:ed:
         fd:00:a7:11:0c:68:e0:4f:f7:a0:d8:13:34:43:7f:09:e8:21:
         2a:9f:34:cc:ab:10:49:ba:ff:21:b9:97:57:25:c9:28:21:66:
         1e:e0:23:45:35:20:c6:8f:a8:93:b5:40:7f:e5:c3:73:ce:2a:
         d1:52:0c:9f:36:53:f7:e6:9b:03:18:94:56:9e:7b:34:20:ba:
         98:b9:fa:85:d9:95:7a:46:8e:d1:f1:b5:56:be:4c:30:be:4e:
         2d:28:37:a3:60:5b:91:b8:c5:38:30:94:3a:7f:fa:0b:bb:f0:
         af:fd:e8:67:78:55:42:11:32:71:57:0e:b3:66:17:83:52:76:
         d7:00:4a:24:87:a0:82:8e:7f:0e:21:64:a8:48:65:82:74:11:
         15:fa:99:6d:c7:5b:cf:5f:09:0b:2b:5a:a4:71:51:af:99:04:
         1e:49:14:01:fe:70:0d:a0:ec:2a:4e:c4:f5:3a:d6:bc:80:7d:
         6c:9c:53:c4:25:32:c8:8a:6f:b7:5a:61:7f:4c:a1:b2:95:c9:
         57:67:8f:45:81:d2:3d:b9:9e:86:91:61:03:2a:51:16:50:3b:
         d4:21:83:89:98:0f:93:4d:00:82:78:67:62:70:8d:c0:e9:3b:
         77:4b:b7:4d:52:1f:f7:cd:04:5e:e4:02:c2:89:7b:64:79:69:
         da:f4:57:3d:36:0d:9e:b4:fc:fc:71:cb:6d:dc:0d:1e:cc:f9:
         d9:5a:5e:36:e3:5c:56:8a:a2:70:75:c2:b4:70:8b:9f:b0:86:
         04:42:a5:35:5b:47:a4:df:e0:94:3a:5a:32:e6:9c:1f:02:10:
         d4:14:88:de:d2:46:fc:bb:b6:1c:71:90:88:6f:07:cb:5c:be:
         5c:e1:52:42:d6:af:e6:3a:7c:c0:10:6c:94:5b:2f:1c:cd:5a:
         a0:c1:de:8d:79:4c:54:9d:a6:60:21:b1:39:0d:bc:98:c9:54:
         ac:54:55:9b:05:44:c4:af:94:f4:2d:b3:cb:8f:31:d4:bf:0d:
         ea:60:66:92:7b:fa:8c:a6
-----BEGIN X509 CRL-----
MIIDcjCCAVoCAQEwDQYJKoZIhvcNAQEMBQAwgYgxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMV
VGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgUlNBIENl
cnRpZmljYXRpb24gQXV0aG9yaXR5Fw0xOTA4MTgxMzU4MjVaFw0xOTA4MjIxMzU4
MjVaMGswIgIRANjIV8rkvmb5E3Wf3aPV2ysXDTE4MDMxNjEyMjE1NFowIgIRAKbc
LtOIMMwDPSOrzrclZL8XDTE4MDMxNjEyMjE1NFowIQIQMpG5djdbcIoI5TIkJ7vE
NBcNMTgwMzE2MTIyMTU0WqAwMC4wHwYDVR0jBBgwFoAUU3m/WqorSs9UgOHYm8Cd
8rIDZsswCwYDVR0UBAQCAg4sMA0GCSqGSIb3DQEBDAUAA4ICAQBlc9HCsiGDaxCc
DYPnBzJ6iyW/HIQDe/WPwTtOP7MMyWEdvkySXs3+VgcL/1I48RsOrUCn2ChZdsLn
JHcpceVTTC/DZc6zSZxF00oKpXUAH2WFnBbi1zhv6r/TWO9+yxB6yJu75GnmNbQx
iBkp3CuHlU+eFwr1FrdwHyp80O39AKcRDGjgT/eg2BM0Q38J6CEqnzTMqxBJuv8h
uZdXJckoIWYe4CNFNSDGj6iTtUB/5cNzzirRUgyfNlP35psDGJRWnns0ILqYufqF
2ZV6Ro7R8bVWvkwwvk4tKDejYFuRuMU4MJQ6f/oLu/Cv/ehneFVCETJxVw6zZheD
UnbXAEokh6CCjn8OIWSoSGWCdBEV+pltx1vPXwkLK1qkcVGvmQQeSRQB/nANoOwq
TsT1Ota8gH1snFPEJTLIim+3WmF/TKGylclXZ49FgdI9uZ6GkWEDKlEWUDvUIYOJ
mA+TTQCCeGdicI3A6Tt3S7dNUh/3zQRe5ALCiXtkeWna9Fc9Ng2etPz8cctt3A0e
zPnZWl4241xWiqJwdcK0cIufsIYEQqU1W0ek3+CUOloy5pwfAhDUFIje0kb8u7Yc
cZCIbwfLXL5c4VJC1q/mOnzAEGyUWy8czVqgwd6NeUxUnaZgIbE5DbyYyVSsVFWb
BUTEr5T0LbPLjzHUvw3qYGaSe/qMpg==
-----END X509 CRL-----

Path #2 Root

Root certificate for Path #2 (AddTrust External CA Root):

# openssl x509 -fingerprint -noout -in AddTrustExternalCARoot.crt
SHA1 Fingerprint=02:FA:F3:E2:91:43:54:68:60:78:57:69:4D:F5:E4:5B:68:85:18:68

Download CRL from CA:

# wget http://crl.comodoca.com/AddTrustExternalCARoot.crl

Convert DER to PEM:

# openssl crl -inform DER -in AddTrustExternalCARoot.crl -outform PEM -out AddTrustExternalCARoot.pem

Get the fingerprint of the CRL:

# openssl crl -fingerprint -noout -in AddTrustExternalCARoot.pem
SHA1 Fingerprint=3B:03:0D:23:1C:F5:1F:53:0A:CC:AA:7A:25:BF:EE:D5:3F:80:8A:BB

Print the text version of the CRL:

# openssl crl -in AddTrustExternalCARoot.pem -text
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
        Last Update: Aug 18 13:58:25 2019 GMT
        Next Update: Aug 22 13:58:25 2019 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A

            X509v3 CRL Number:
                5362
Revoked Certificates:
    Serial Number: 537B76564F297F14DC6943E922AD2C79
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 46EAF096054CC5E3FA65EA6E9F42C664
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 3ACDAB9C759886BCAF74E5DF81A9F4E8
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 79174AA9141736FE15A7CA9F2CFF4588
        Revocation Date: Apr 30 20:03:54 2018 GMT
    Serial Number: 74C18753F7EEB4EA238D8416B5AC7646
        Revocation Date: Oct  9 09:11:57 2018 GMT
    Signature Algorithm: sha1WithRSAEncryption
         a8:f8:d9:b4:e2:75:46:19:27:0f:88:68:69:fa:06:f9:6a:51:
         bd:17:6e:8f:35:7c:2d:27:28:84:b1:59:99:c5:59:12:60:bc:
         50:f6:32:47:5b:f7:72:b9:42:40:2a:00:e5:63:e0:af:45:04:
         99:72:6c:ee:30:e7:dd:53:f6:1c:7b:9e:01:b5:06:ea:86:5f:
         8a:66:65:12:37:75:55:8a:3b:91:dd:87:43:be:c4:ce:6c:29:
         80:33:38:f4:df:e6:ed:d3:c9:ea:5c:e6:ee:57:22:7e:ca:fe:
         58:bd:e9:1d:f6:1a:02:9c:a1:86:77:ed:c1:7b:ea:a1:c8:cf:
         0f:5f:5b:ad:92:38:fc:86:45:5f:fc:99:cb:19:25:62:b0:61:
         6b:a7:8b:70:71:38:5a:39:e6:7f:de:d1:84:db:1c:cc:f3:e2:
         88:c6:4f:89:7a:5b:04:3d:cf:71:51:60:41:6d:38:9a:3f:08:
         4b:35:00:63:87:97:70:1f:15:ff:e1:72:20:7b:59:1f:de:41:
         e4:81:6f:26:12:5e:c6:6f:cb:77:00:99:97:da:c2:68:fe:d5:
         07:3f:a5:e0:fa:33:6b:c9:f3:b8:7f:02:05:b4:23:9f:4b:5d:
         5c:2f:dd:81:7c:bf:5c:3f:87:f1:dd:03:03:f3:bc:3e:68:86:
         ae:a9:64:7d
-----BEGIN X509 CRL-----
MIICnTCCAYUCAQEwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0UxFDASBgNV
BAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAg
TmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdBcNMTkw
ODE4MTM1ODI1WhcNMTkwODIyMTM1ODI1WjCBrzAhAhBTe3ZWTyl/FNxpQ+kirSx5
Fw0xNTEyMTQxNTU4MzBaMCECEEbq8JYFTMXj+mXqbp9CxmQXDTE1MTIxNDE1NTgz
MFowIQIQOs2rnHWYhryvdOXfgan06BcNMTUxMjE0MTU1ODMwWjAhAhB5F0qpFBc2
/hWnyp8s/0WIFw0xODA0MzAyMDAzNTRaMCECEHTBh1P37rTqI42EFrWsdkYXDTE4
MTAwOTA5MTE1N1qgMDAuMB8GA1UdIwQYMBaAFK29mHo0tCb3+sQmVO8DveAky1Qa
MAsGA1UdFAQEAgIU8jANBgkqhkiG9w0BAQUFAAOCAQEAqPjZtOJ1RhknD4hoafoG
+WpRvRdujzV8LScohLFZmcVZEmC8UPYyR1v3crlCQCoA5WPgr0UEmXJs7jDn3VP2
HHueAbUG6oZfimZlEjd1VYo7kd2HQ77EzmwpgDM49N/m7dPJ6lzm7lcifsr+WL3p
HfYaApyhhnftwXvqocjPD19brZI4/IZFX/yZyxklYrBha6eLcHE4Wjnmf97RhNsc
zPPiiMZPiXpbBD3PcVFgQW04mj8ISzUAY4eXcB8V/+FyIHtZH95B5IFvJhJexm/L
dwCZl9rCaP7VBz+l4Poza8nzuH8CBbQjn0tdXC/dgXy/XD+H8d0DA/O8PmiGrqlk
fQ==
-----END X509 CRL-----

CRL from CA (Path #2 Root, older CRL)

And here is the CRL supplied by the intermediate CA that signed the wildcard certificate:

# openssl crl -fingerprint -noout -in root.crl
SHA1 Fingerprint=E1:24:F4:35:3B:D0:B7:5B:AA:D9:AD:C7:33:F2:29:32:20:08:47:14

Here, I note that the above CRL fingerprint is different, but the X509v3 Authority Key Identifier is an exact match (this is an older version of the above CRL).

# openssl crl -in root.crl -text
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
        Last Update: May 28 00:12:38 2019 GMT
        Next Update: Jun  1 00:12:38 2019 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A

            X509v3 CRL Number:
                5275
Revoked Certificates:
    Serial Number: 537B76564F297F14DC6943E922AD2C79
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 46EAF096054CC5E3FA65EA6E9F42C664
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 3ACDAB9C759886BCAF74E5DF81A9F4E8
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 79174AA9141736FE15A7CA9F2CFF4588
        Revocation Date: Apr 30 20:03:54 2018 GMT
    Serial Number: 74C18753F7EEB4EA238D8416B5AC7646
        Revocation Date: Oct  9 09:11:57 2018 GMT
    Signature Algorithm: sha1WithRSAEncryption
         38:3a:7d:3e:ee:be:48:e7:93:c3:91:0a:c3:47:46:11:87:83:
         60:85:19:2f:77:17:bd:e9:0f:02:de:04:60:33:38:5a:38:99:
         16:6b:81:51:83:f4:dc:97:83:2f:f9:97:18:5e:6a:24:5e:77:
         a0:39:dc:e1:09:24:c8:9c:05:e3:68:a2:ca:aa:1f:e7:85:fb:
         84:a0:07:96:4d:f0:53:68:6f:85:bd:d6:07:6d:57:34:9a:01:
         6e:51:b5:53:69:da:db:e1:c6:0d:c6:d9:d6:85:96:2e:b0:bf:
         71:25:49:97:66:8b:61:4a:7d:fd:ce:f3:07:d2:b5:bf:71:c0:
         01:6b:79:b2:20:5f:58:41:34:03:1b:88:a5:d7:f7:9b:ab:ff:
         49:fa:07:0b:0a:90:d8:f8:93:28:70:7a:f9:48:ed:40:b3:ae:
         31:f5:af:51:ed:00:ff:2e:0e:b9:3e:ee:6c:20:21:a0:d8:98:
         46:bd:9d:00:bb:aa:3d:30:8c:b2:72:00:af:cd:79:05:2f:40:
         5a:ae:2a:27:26:77:c3:40:79:88:4c:7b:2e:2e:df:2e:d5:4f:
         c5:b2:14:e1:aa:9a:29:4f:b5:e5:01:04:df:b7:89:59:17:1c:
         06:7b:a1:a8:9a:84:0c:cf:8c:2f:e9:3a:ec:78:f5:c9:e2:da:
         5f:16:1a:38
-----BEGIN X509 CRL-----
MIICnTCCAYUCAQEwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0UxFDASBgNV
BAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAg
TmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdBcNMTkw
NTI4MDAxMjM4WhcNMTkwNjAxMDAxMjM4WjCBrzAhAhBTe3ZWTyl/FNxpQ+kirSx5
Fw0xNTEyMTQxNTU4MzBaMCECEEbq8JYFTMXj+mXqbp9CxmQXDTE1MTIxNDE1NTgz
MFowIQIQOs2rnHWYhryvdOXfgan06BcNMTUxMjE0MTU1ODMwWjAhAhB5F0qpFBc2
/hWnyp8s/0WIFw0xODA0MzAyMDAzNTRaMCECEHTBh1P37rTqI42EFrWsdkYXDTE4
MTAwOTA5MTE1N1qgMDAuMB8GA1UdIwQYMBaAFK29mHo0tCb3+sQmVO8DveAky1Qa
MAsGA1UdFAQEAgIUmzANBgkqhkiG9w0BAQUFAAOCAQEAODp9Pu6+SOeTw5EKw0dG
EYeDYIUZL3cXvekPAt4EYDM4WjiZFmuBUYP03JeDL/mXGF5qJF53oDnc4QkkyJwF
42iiyqof54X7hKAHlk3wU2hvhb3WB21XNJoBblG1U2na2+HGDcbZ1oWWLrC/cSVJ
l2aLYUp9/c7zB9K1v3HAAWt5siBfWEE0AxuIpdf3m6v/SfoHCwqQ2PiTKHB6+Ujt
QLOuMfWvUe0A/y4OuT7ubCAhoNiYRr2dALuqPTCMsnIAr815BS9AWq4qJyZ3w0B5
iEx7Li7fLtVPxbIU4aqaKU+15QEE37eJWRccBnuhqJqEDM+ML+k67Hj1yeLaXxYa
OA==
-----END X509 CRL-----

Parker

Asked: 2019-08-15 08:55:51 +0800 CST

Errors when attempting to connect to PostgreSQL 9.6 using SSL wildcard server certificate and no client certificates

3

I have a PostgreSQL 9.6.11 database on Amazon Linux that has been configured with a 2048-bit SSL wildcard server certificate and password-based (no client certificates) remote connections since January 2012. After a recent certificate upgrade (Comodo, now Sectigo), I can no longer establish remote psql or JDBC connections to this database over SSL.

My goal is to be able to connect to this PostgreSQL database remotely via psql and also via JDBC.

Starting from the server key (which hasn't changed since I had remote access working), I have attempted to cover the full series of steps to verify that I have my keys, certificates, firewall and database set up correctly.

I must have missed something, since I'm unable to connect remotely via psql or JDBC.

What have I missed that could be causing these remote connections to fail?

Troubleshooting Steps

As user postgres:

# cd /var/lib/pgsql96/data

postgresql.conf

I had tried to limit the set of ciphers to attempt to force TLSv1.2 for all SSL connections. Since there was no difference in client behavior, I commented out ssl_ciphers and ssl_prefer_server_ciphers to allow the defaults.

ssl = on
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1'
#ssl_prefer_server_ciphers = on
ssl_cert_file = 'server.crt' # wildcard cert plus intermediate certs
ssl_key_file = 'server.key' # private key
#ssl_ca_file = 'root.crt' # commented out - do not require client certs
#ssl_crl_file = 'root.crl' # commented out - no client certificates

pg_hba.conf

This file was set up to allow only the public IP address of the localhost and the remote host I am testing. I don't want to require client certificates, only encryption with a required password.

hostssl       all     all     11.222.11.222/32      password # localhost
hostssl       all     all     34.84.31.82/32        password # remote host

I checked the certification paths via ssltest and found that there are two paths available (Path #1 and Path #2):

From the documentation on PostgreSQL 9.6 Secure TCP/IP Connections with SSL:

The first certificate in server.crt must be the server's certificate because it must match the server's private key. The certificates of "intermediate" certificate authorities can also be appended to the file. Doing this avoids the necessity of storing intermediate certificates on clients, assuming the root and intermediate certificates were created with v3_ca extensions. This allows easier expiration of intermediate certificates.

It is not necessary to add the root certificate to server.crt. Instead, clients must have the root certificate of the server's certificate chain.

Assembling and verifying the certificate chain for Path #1

# ls -l
-rw------- 1 postgres postgres 2313 Aug 15 00:26 1_wildcard_server.crt
-rw------- 1 postgres postgres 2167 Aug 15 00:27 2_intermediate_sectigo.crt
-rw------- 1 postgres postgres 2094 Aug 15 00:27 3_root_usertrust-selfsigned.crt

I checked the fingerprints of each individual certificate in Path #1 to confirm their identity:

# openssl x509 -in 1_wildcard_server.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=8C:69:06:8E:81:31:30:6E:DA:DD:C2:1C:38:83:73:67:97:3D:DB:37:78:B8:49:D7:7E:32:A8:3F:1F:8B:08:AB

# openssl x509 -in 2_intermediate_sectigo.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=7F:A4:FF:68:EC:04:A9:9D:75:28:D5:08:5F:94:90:7F:4D:1D:D1:C5:38:1B:AC:DC:83:2E:D5:C9:60:21:46:76

# openssl x509 -in 3_root_usertrust-selfsigned.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=E7:93:C9:B0:2F:D8:AA:13:E2:1C:31:22:8A:CC:B0:81:19:64:3B:74:9C:89:89:64:B1:74:6D:46:C3:D4:CB:D2

And also examined the text versions of the certificates to confirm that the intermediate and root certificates have the v3_ca extension (the wildcard server certificate does not have this extension):

# openssl x509 -in 1_wildcard_server.crt -text
...
X509v3 Basic Constraints: critical
    CA:FALSE
...

# openssl x509 -in 2_intermediate_sectigo.crt -text
...
X509v3 Basic Constraints: critical
    CA:TRUE
...

# openssl x509 -in 3_root_usertrust-selfsigned.crt -text
...
X509v3 Basic Constraints: critical
    CA:TRUE
...

Verify the certificate chain for Path #1 with OpenSSL:

# openssl verify -verbose -CAfile 3_root_usertrust-selfsigned.crt -untrusted 2_intermediate_sectigo.crt 1_wildcard_server.crt
1_wildcard_server.crt: OK

Create the bundled server and intermediate certificate for Path #1:

# cat 1_wildcard_server.crt > server.crt
# cat 2_intermediate_sectigo.crt >> server.crt

Create the bundled intermediate and root certificates (those with the v3_ca extension) for Path #1 (although this would only be needed when requiring client certificates):

# cat 2_intermediate_sectigo.crt > root.crt
# cat 3_root_usertrust-selfsigned.crt >> root.crt

Assembling and verifying the certificate chain for Path #2

# ls -l
-rw------- 1 postgres postgres 2313 Aug 15 00:26 1_wildcard_server.crt
-rw------- 1 postgres postgres 2167 Aug 15 00:27 2_intermediate_sectigo.crt
-rw------- 1 postgres postgres 1956 Aug 15 00:35 3_intermediate_usertrust.crt
-rw------- 1 postgres postgres 1521 Aug 15 00:27 4_root_addtrustroot-selfsigned.crt

I checked the fingerprints of each individual certificate in Path #2 to confirm their identity:

# openssl x509 -in 1_wildcard_server.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=8C:69:06:8E:81:31:30:6E:DA:DD:C2:1C:38:83:73:67:97:3D:DB:37:78:B8:49:D7:7E:32:A8:3F:1F:8B:08:AB

# openssl x509 -in 2_intermediate_sectigo.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=7F:A4:FF:68:EC:04:A9:9D:75:28:D5:08:5F:94:90:7F:4D:1D:D1:C5:38:1B:AC:DC:83:2E:D5:C9:60:21:46:76

# openssl x509 -in 3_intermediate_usertrust.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=1A:51:74:98:0A:29:4A:52:8A:11:07:26:D5:85:56:50:26:6C:48:D9:88:3B:EA:69:2B:67:B6:D7:26:DA:98:C5

# openssl x509 -in 4_root_addtrustroot-selfsigned.crt -noout -sha256 -fingerprint
SHA256 Fingerprint=68:7F:A4:51:38:22:78:FF:F0:C8:B1:1F:8D:43:D5:76:67:1C:6E:B2:BC:EA:B4:13:FB:83:D9:65:D0:6D:2F:F2

And also examined the text versions of the certificates to confirm that the intermediate and root certificates have the v3_ca extension (the wildcard server certificate does not have this extension):

# openssl x509 -in 1_wildcard_server.crt -text
...
X509v3 Basic Constraints: critical
    CA:FALSE
...

# openssl x509 -in 2_intermediate_sectigo.crt -text
...
X509v3 Basic Constraints: critical
    CA:TRUE
...

# openssl x509 -in 3_intermediate_usertrust.crt -text
...
X509v3 Basic Constraints: critical
    CA:TRUE
...

# openssl x509 -in 4_root_addtrustroot-selfsigned.crt -text
...
X509v3 Basic Constraints: critical
    CA:TRUE
...

Verify the certificate chain for Path #2 with OpenSSL (from this article):

# openssl verify -verbose -CAfile 4_root_addtrustroot-selfsigned.crt -untrusted 3_intermediate_usertrust.crt 2_intermediate_sectigo.crt
2_intermediate_sectigo.crt: OK

Create the bundled server and intermediate certificate for Path #2:

# cat 1_wildcard_server.crt > server.crt
# cat 2_intermediate_sectigo.crt >> server.crt
# cat 3_intermediate_usertrust.crt >> server.crt

Create the bundled intermediate and root certificates (those with the v3_ca extension) for Path #2 (although this would only be needed when requiring client certificates):

# cat 2_intermediate_sectigo.crt > root.crt
# cat 3_intermediate_usertrust.crt >> root.crt
# cat 4_root_addtrustroot-selfsigned.crt >> root.crt

Verified key and certificate permissions (as in this question).

# ls -l *.key *.crt *.crl-bash-4.2$ ls -l *.key *.crt *.crl
-rw-r----- 1 postgres postgres  963 Aug 14 21:12 root.crl
-rw-r--r-- 1 postgres postgres 1521 Aug 15 01:27 root.crt
-rw-r--r-- 1 postgres postgres 6436 Aug 15 01:27 server.crt
-rw------- 1 postgres postgres 1679 May 28 19:33 server.key

Confirmed that the server key is ok, following instructions from Comodo.

# openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
# openssl rsa -check -noout -in server.key
RSA key ok

Confirmed that the certificate and private key moduli are identical.

# openssl rsa -modulus -noout -in server.key
Modulus=[REDACTED]

# openssl x509 -modulus -noout -in server.crt
Modulus=[REDACTED]

Tested the CRL and verified the issuer:

# openssl crl -in root.crl -text
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
        Last Update: May 28 00:12:38 2019 GMT
        Next Update: Jun  1 00:12:38 2019 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            X509v3 CRL Number:
                5275
Revoked Certificates:
    Serial Number: 537B76564F297F14DC6943E922AD2C79
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 46EAF096054CC5E3FA65EA6E9F42C664
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 3ACDAB9C759886BCAF74E5DF81A9F4E8
        Revocation Date: Dec 14 15:58:30 2015 GMT
    Serial Number: 79174AA9141736FE15A7CA9F2CFF4588
        Revocation Date: Apr 30 20:03:54 2018 GMT
    Serial Number: 74C18753F7EEB4EA238D8416B5AC7646
        Revocation Date: Oct  9 09:11:57 2018 GMT
    Signature Algorithm: sha1WithRSAEncryption
         38:3a:7d:3e:ee:be:48:e7:93:c3:91:0a:c3:47:46:11:87:83:
[TRIMMED]
         5f:16:1a:38
-----BEGIN X509 CRL-----
MIICnTCCAYUCAQEwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0UxFDASBgNV
[TRIMMED]
iEx7Li7fLtVPxbIU4aqaKU+15QEE37eJWRccBnuhqJqEDM+ML+k67Hj1yeLaXxYa
OA==
-----END X509 CRL-----

Started PostgreSQL service.

# service postgresql96 start
Starting postgresql96 service:                             [  OK  ]

Verified there were no errors in the log files.

# cat ../pgstartup.log
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".

# cat pg_log/postgresql-Wed.log
LOG:  database system was shut down at 2019-08-14 15:01:03 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

Successfully connected to database via SSL from localhost with sslmode require and with verify-full (using FQDN).

# psql "postgresql://mydbuser@localhost:5432/mydb?ssl=true&sslmode=require"
psql (9.6.11)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
mydb=> \q

# psql "postgresql://mydbuser@[REDACTED].org:5432/mydb?ssl=true&sslmode=verify-full"
Password: ********
psql (9.6.11)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
mydb=> \q

Retrieved and visually verified the SSL certificate chain remotely via Java using a combination of this method, this method, this method and this method. This confirms that port 5432 is open for this host on the firewall and that the certificate and chain are retrievable via JDBC.

Supported Protocols: 5
 SSLv2Hello
 SSLv3
 TLSv1
 TLSv1.1
 TLSv1.2
Enabled Protocols: 3
 TLSv1
 TLSv1.1
 TLSv1.2
Enabled Cipher suites: 43
 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 TLS_RSA_WITH_AES_256_CBC_SHA256
 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 TLS_RSA_WITH_AES_256_CBC_SHA
 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 TLS_RSA_WITH_AES_128_CBC_SHA256
 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 TLS_RSA_WITH_AES_128_CBC_SHA
 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 TLS_RSA_WITH_AES_256_GCM_SHA384
 TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
 TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
 TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 TLS_RSA_WITH_AES_128_GCM_SHA256
 TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
 TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
 TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
 TLS_EMPTY_RENEGOTIATION_INFO_SCSV
Cert 1 (active):
    Thumbprint SHA1   : B5:2D:43:A8:0F:C6:C3:39:1F:2D:BB:9C:30:A5:4B:8D:DF:5F:9B:F8
    Fingerprint SHA256: 8c69068e8131306edaddc21c38837367973ddb3778b849d77e32a83f1f8b08ab
    Subject: CN=*.[REDACTED].org,OU=PositiveSSL Wildcard,OU=Domain Control Validated
    Issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
Cert 2 (active):
    Thumbprint SHA1   : 33:E4:E8:08:07:20:4C:2B:61:82:A3:A1:4B:59:1A:CD:25:B5:F0:DB
    Fingerprint SHA256: 7fa4ff68ec04a99d7528d5085f94907f4d1dd1c5381bacdc832ed5c960214676
    Subject: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
    Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
Cert 3 (active):
    Thumbprint SHA1   : 33:E4:E8:08:07:20:4C:2B:61:82:A3:A1:4B:59:1A:CD:25:B5:F0:DB
    Fingerprint SHA256: 7fa4ff68ec04a99d7528d5085f94907f4d1dd1c5381bacdc832ed5c960214676
    Subject: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
    Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
Cert 4 (active):
    Thumbprint SHA1   : EA:B0:40:68:9A:0D:80:5B:5D:6F:D6:54:FC:16:8C:FF:00:B7:8B:E3
    Fingerprint SHA256: 1a5174980a294a528a110726d5855650266c48d9883bea692b67b6d726da98c5
    Subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
    Issuer: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE

I confirmed that the above root certificates (AddTrust External CA Root) are both in the default Java keystore as recommended here (and also confirmed that they are in the Windows keystore by default as well):

C:\Windows\System32>"C:\Program Files\Java\jdk1.8.0_212\jre\bin\keytool.exe" -keystore "C:\Program Files\Java\jdk1.8.0_212\jre\lib\security\cacerts" -storepass
changeit -list
Keystore type: jks
Keystore provider: SUN

Your keystore contains 95 entries
....
usertrustrsaca [jdk], Aug 25, 2016, trustedCertEntry,
Certificate fingerprint (SHA1): 2B:8F:1B:57:33:0D:BB:A2:D0:7A:6C:51:F7:0E:E9:0D:DA:B9:AD:8E
....
addtrustexternalca [jdk], Aug 25, 2016, trustedCertEntry,
Certificate fingerprint (SHA1): 02:FA:F3:E2:91:43:54:68:60:78:57:69:4D:F5:E4:5B:68:85:18:68
....

Attempted to connect to PostgreSQL via openssl from remote host (as in this question).

# openssl version
OpenSSL 1.1.0h  27 Mar 2018
# openssl s_client -connect [REDACTED].org:5432 -state -msg -showcerts -debug
CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x2070760 [0x20fe520] (289 bytes => 289 (0x121))
0000 - 16 03 01 01 1c 01 00 01-18 03 03 0c 53 44 0c a3   ............SD..
[TRIMMED]
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0120 - 01                                                .
>>> TLS 1.2  [length 0005]
    16 03 01 01 1c
>>> TLS 1.2 Handshake [length 011c], ClientHello
    01 00 01 18 03 03 0c 53 44 0c a3 e2 21 36 f2 b0
[TRIMMED]
    01 05 02 05 03 04 01 04 02 04 03 03 01 03 02 03
    03 02 01 02 02 02 03 00 0f 00 01 01
SSL_connect:SSLv2/v3 write client hello A
read from 0x2070760 [0x2103a80] (7 bytes => 0 (0x0))
139650021263184:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1565797370
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

# tail pg_log/postgresql-Wed.log
LOG:  invalid length of startup packet

Attempted to connect remotely via Windows psql client (9.6.5).

C:\Program Files\PostgreSQL\9.6\bin>psql "postgresql://mydbuser@[REDACTED].org:5432/mydb?ssl=true&sslmode=require"
psql: SSL error: certificate verify failed

For some reason, the psql client is sending an TLSv1 alert, "unknown ca":

# tail pg_log/postgresql-Wed.log
LOG:  could not accept SSL connection: tlsv1 [alert][15] unknown ca

Additionally, if I specify the root certificate:

C:\Program Files\PostgreSQL\9.6\bin>psql "postgresql://mydbuser@[REDACTED].org:5432/mydb?ssl=true&sslmode=require&sslrootcert=root.crt"
psql: SSL error: certificate verify failed

or if I leave that parameter blank:

C:\Program Files\PostgreSQL\9.6\bin>psql "postgresql://mydbuser@[REDACTED].org:5432/mydb?ssl=true&sslmode=require&sslrootcert="
psql: SSL error: certificate verify failed

However, if I specify a non-existent file for that parameter, I am able to connect successfully (with sslmode=require):

C:\Program Files\PostgreSQL\9.6\bin>psql "postgresql://mydbuser@[REDACTED].org:5432/mydb?ssl=true&sslmode=require&sslrootcert=x"
Password:
psql (9.6.5, server 9.6.11)
WARNING: Console code page (437) differs from Windows code page (1252)
         8-bit characters might not work correctly. See psql reference
         page "Notes for Windows users" for details.
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
n4l_live=> \q

From the specification:

unknown_ca

  A valid certificate chain or partial chain was received, but the
  certificate was not accepted because the CA certificate could not
  be located or couldn't be matched with a known, trusted CA.  This
  message is always fatal.

Attempt to connect via Java client with postgresql-42.2.5.jar.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
  at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
  at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
  at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
  at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
  at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:40)
  ... 36 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
  at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
  at sun.security.validator.Validator.validate(Validator.java:262)
  at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
  at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
  at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
  ... 44 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
  at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
  at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
  ... 50 more

For some reason, the Java client is producing an SSLv3 alert, "certificate unknown", even though it is not one of the enabled protocols:

# tail pg_log/postgresql-Wed.log
LOG:  could not accept SSL connection: sslv3 alert certificate unknown

From the specification:

certificate_unknown

  Some other (unspecified) issue arose in processing the
  certificate, rendering it unacceptable.

The PostgreSQL JDBC driver documentation indicates that the Java client (by default) will attempt to use verify-full for the server certificate, which may be the reason a different error is produced here than with the psql client:

The simplest being ssl=true, passing this into the driver will cause the driver to validate both the SSL certificate and verify the hostname (same as verify-full). Note this is different than libpq which defaults to a non-validating SSL connection.

The above failures (and the success when specifying a non-existent root certificate with sslmode=require) seem to point to some problem with verifying the CA for the certificate.

Note: I used the script provided in this answer to generate new certificates and performed the above tests again, with identical results.

Parker

Asked: 2019-06-25 08:40:45 +0800 CST

Sporadic mod_jk Client Errors when uploading large files

0

Note: There is a TL;DR at the end of this question.

I have a nightly backup script that has been working reliably for 6 years, and two months ago, it began to fail to upload to my archive server (only about 1 in 4 attempts succeed). This started to happen once the size of the backup file surpassed about 440MB (successful uploads take 20-40 seconds to complete). Other, smaller files (>200MB) are uploading successfully every time. The start of the failures did not coincide with any configuration changes or updates to the servers in question.

Here are the details of the file I am uploading:

-rw-r--r-- 1 backup backup 486948951 Jun 12 13:20 snapshot-20190612.zip

These files are uploaded to my server with a curl PUT (see command below). The source server is an Amazon Linux instance in Virginia, and the destination server is an Amazon Linux instance in the Ohio (both are in the US East region).

The web service that handles the upload is an Apache HTTPD 2.4.39 + mod24_ssl/OpenSSL 1.0.2k -> mod_jk 1.2.42 -> Apache Tomcat 8.5.40 -> Artifactory 5.3.2 instance.

I captured the jk_status page before and after one of these errors occurs (before/after are in an animated GIF to highlight the differences). The error is being interpreted as a Client Error by mod_jk.

My mod/jk workers.properties file:

worker.list=main_ajp13_worker
# Set properties for main_ajp13_worker (ajp13)
worker.main_ajp13_worker.type=ajp13
worker.main_ajp13_worker.host=localhost
worker.main_ajp13_worker.port=8009

The following are the console output (authorization details scrubbed) from a failed upload, followed immediately by a successful upload:

-bash-4.2$ curl -v -k -u backup:PASSWORD -H X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3 -H X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10 -T snapshot-20190612.zip https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
*   Trying 123.222.100.132...
* TCP_NODELAY set
* Connected to repo.mydomain.org (123.222.100.132) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.mydomain.org
*  start date: May 28 00:00:00 2019 GMT
*  expire date: May 27 23:59:59 2021 GMT
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'backup'
> PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1
> Host: repo.mydomain.org
> Authorization: Basic DELETED_TOKEN
> User-Agent: curl/7.61.1
> Accept: */*
> X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3
> X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10
> Content-Length: 486948951
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* Closing connection 0
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

-bash-4.2$ curl -v -u backup:PASSWORD -H X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3 -H X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10 -T snapshot-20190612.zip https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
*   Trying 123.222.100.132...
* TCP_NODELAY set
* Connected to repo.mydomain.org (123.222.100.132) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.mydomain.org
*  start date: May 28 00:00:00 2019 GMT
*  expire date: May 27 23:59:59 2021 GMT
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'backup'
> PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1
> Host: repo.mydomain.org
> Authorization: Basic DELETED_TOKEN
> User-Agent: curl/7.61.1
> Accept: */*
> X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3
> X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10
> Content-Length: 486948951
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* Closing connection 0
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

With a second (or often third or fourth) attempt of the upload, it eventually succeeds:

-bash-4.2$ curl -v -u backup:PASSWORD -H X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3 -H X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10 -T snapshot-20190612.zip https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
*   Trying 123.222.100.132...
* TCP_NODELAY set
* Connected to repo.mydomain.org (123.222.100.132) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.mydomain.org
*  start date: May 28 00:00:00 2019 GMT
*  expire date: May 27 23:59:59 2021 GMT
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'backup'
> PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1
> Host: repo.mydomain.org
> Authorization: Basic DELETED_TOKEN
> User-Agent: curl/7.61.1
> Accept: */*
> X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3
> X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10
> Content-Length: 486948951
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 201
< Date: Wed, 12 Jun 2019 13:22:46 GMT
< Server: Apache
< Strict-Transport-Security: max-age=15768000
< X-Artifactory-Id: 9e49bf90cd4b7700:7ba45cf1:16b1db56e4d:-8000
< Location: https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
< Transfer-Encoding: chunked
< Content-Type: application/vnd.org.jfrog.artifactory.storage.itemcreated+json;charset=ISO-8859-1
<
{
  "repo" : "resource-snapshot",
  "path" : "/snapshot/20190612/snapshot-20190612.zip",
  "created" : "2019-06-12T13:23:05.805Z",
  "createdBy" : "backup",
  "downloadUri" : "https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip",
  "mimeType" : "application/zip",
  "size" : "486948951",
  "checksums" : {
    "sha1" : "6b9525bf9bb572ec7f70d0d474cb6533e0091bd3",
    "md5" : "6da0c444ad1b53dc1cf06784e204cb10"
  },
  "originalChecksums" : {
    "sha1" : "6b9525bf9bb572ec7f70d0d474cb6533e0091bd3",
    "md5" : "6da0c444ad1b53dc1cf06784e204cb10"
  },
  "uri" : "https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip"
* Connection #0 to host repo.mydomain.org left intact
}

The server logs aren't particularly informative. Here are the relevant logs when an upload fails:

/var/log/httpd/ssl_request_log

[24/Jun/2019:12:17:45 +0000] 52.1.100.190 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1" -

/var/log/httpd/repo.mydomain.org-ssl_access_log

140.1.100.101 - - [24/Jun/2019:12:17:45 +0000] "PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1" 400 -

catalina.out

No log entries.

/var/log/httpd/mod_jk.log

These are the logs when the mod/jk log level is set to info:

[Mon Jun 24 12:18:05 2019] [15679:140534117611584] [info] ajp_read_into_msg_buff::jk_ajp_common.c (1553): (main_ajp13_worker) receiving data from client failed. Connection aborted or network problems
[Mon Jun 24 12:18:05 2019] [15679:140534117611584] [info] ajp_process_callback::jk_ajp_common.c (2105): (main_ajp13_worker) Reading from client aborted or client network problems
[Mon Jun 24 12:18:05 2019] [15679:140534117611584] [info] ajp_service::jk_ajp_common.c (2778): (main_ajp13_worker) sending request to tomcat failed (unrecoverable), because of client read error (attempt=1)
[Mon Jun 24 12:18:05 2019] [15679:140534117611584] [info] jk_handler::mod_jk.c (2988): Aborting connection for worker=main_ajp13_worker

And here is the relevant part of the 2.4 million mod_jk log entries at the debug level, starting right at the HTTP 100 Continue and showing the very start of the file upload (you can see the magic number of the zip file, 50 4B 03 04):

[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0160    32 65 32 00 A0 08 00 09 34 39 36 32 37 38 36 30  - 2e2.....49627860
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0170    33 00 00 06 45 78 70 65 63 74 00 00 0C 31 30 30  - 3...Expect...100
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0180    2D 63 6F 6E 74 69 6E 75 65 00 0A 00 0F 41 4A 50  - -continue....AJP
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0190    5F 52 45 4D 4F 54 45 5F 50 4F 52 54 00 00 05 33  - _REMOTE_PORT...3
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 01a0    32 39 37 30 00 0A 00 0E 41 4A 50 5F 4C 4F 43 41  - 2970....AJP_LOCA
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 01b0    4C 5F 41 44 44 52 00 00 0C 31 37 32 2E 33 31 2E  - L_ADDR...172.31.
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 01c0    31 33 2E 39 36 00 0A 00 10 4A 4B 5F 4C 42 5F 41  - 13.96....JK_LB_A
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 01d0    43 54 49 56 41 54 49 4F 4E 00 00 03 41 43 54 00  - CTIVATION...ACT.
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 01e0    0A 00 0F 53 53 4C 5F 43 4C 49 45 4E 54 5F 43 45  - ...SSL_CLIENT_CE
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 01f0    52 54 00 00 0F 53 53 4C 5F 43 4C 49 45 4E 54 5F  - RT...SSL_CLIENT_
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0200    43 45 52 54 00 FF 00 00 00 00 00 00 00 00 00 00  - CERT............
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_send_request::jk_ajp_common.c (1782): (main_ajp13_worker) request body to send 496278603 - request body to resend 0
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_send_request::jk_ajp_common.c (1886): (main_ajp13_worker) sending 8186 bytes of request body
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): sending to ajp13 pos=4 len=8192 max=8192
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0000    12 34 1F FC 1F FA 50 4B 03 04 14 00 00 00 08 00  - .4....PK........
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0010    84 61 D8 4E 73 67 93 AF 7C 0A 00 00 E0 B0 00 00  - .a.Nsg..|.......
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0020    0A 00 1C 00 62 61 63 6B 75 70 2E 6C 6F 67 55 54  - ....backup.logUT
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0030    09 00 03 18 BE 10 5D 90 BD 10 5D 75 78 0B 00 01  - ......]...]ux...
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0040    04 1A 00 00 00 04 1A 00 00 00 ED 5C 4D 6F DC C8  - ...........\Mo..
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0050    11 BD E7 57 34 74 4A 80 D8 80 77 17 8B 45 80 1C  - ...W4tJ...w..E..

.... [lots of encoded data trimmed out here, the next part shows re-filling the 8k buffer] ....

[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1462): received from ajp13 pos=0 len=3 max=8192E D7 E9 E6 DD BB  - 8...U?.er.>.....
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1462): 00000   06 1F FA 00 00 00 00 00 00 00 00 00 00 00 00 00  - .................
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): sending to ajp13 pos=4 len=8192 max=8192 2E 40 41 4C 94  - ..$z~YW.?1G.@AL.
[Mon Jun 24 14:37:08 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 0000    12 34 1F FC 1F FA C9 62 3E 48 C5 70 E9 05 E1 8D  - .4.....b>H.p....

.... [lots of encoded data trimmed out here, the next part shows where re-filling the 8k buffer fails] ....

[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 03e0    2D E5 36 F3 8C 12 8C 51 69 66 21 AE A5 58 BE 6E  - -.6....Qif!..X.n
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1267): 03f0    57 F3 F4 EC CB E9 49 9B 59 44 AB FE 66 6E 67 DC  - W.....I.YD..fng.
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1462): received from ajp13 pos=0 len=3 max=8192
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1462): 0000   06 1F FA 00 00 00 00 00 00 00 00 00 00 00 00 00  - ................
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [info] ajp_read_into_msg_buff::jk_ajp_common.c (1553): (main_ajp13_worker) receiving data from client failed. Connection aborted or network problems
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [info] ajp_process_callback::jk_ajp_common.c (2105): (main_ajp13_worker) Reading from client aborted or client network problems
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [info] ajp_service::jk_ajp_common.c (2778): (main_ajp13_worker) sending request to tomcat failed (unrecoverable), because of client read error (attempt=1)
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_reset_endpoint::jk_ajp_common.c (851): (main_ajp13_worker) resetting endpoint with socket 24 (socket shutdown)
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_abort_endpoint::jk_ajp_common.c (821): (main_ajp13_worker) aborting endpoint with socket 24
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] jk_shutdown_socket::jk_connect.c (932): About to shutdown socket 24 [127.0.0.1:38678 -> 127.0.0.1:8009]
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] jk_shutdown_socket::jk_connect.c (1005): shutting down the read side of socket 24 [127.0.0.1:38678 -> 127.0.0.1:8009]
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] jk_shutdown_socket::jk_connect.c (1016): Shutdown socket 24 [127.0.0.1:38678 -> 127.0.0.1:8009] and read 265 lingering bytes in 0 sec.
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [debug] ajp_done::jk_ajp_common.c (3287): recycling connection pool for worker main_ajp13_worker and socket -1
[Mon Jun 24 14:37:28 2019] [17826:139891516274752] [info] jk_handler::mod_jk.c (2988): Aborting connection for worker=main_ajp13_worker
[Mon Jun 24 14:38:24 2019] [17825:139891516274752] [debug] wc_shutdown::jk_worker.c (390): Shutting down worker main_ajp13_worker
[Mon Jun 24 14:38:24 2019] [17825:139891516274752] [debug] jk_shm_close::jk_shm.c (745): Closed shared memory /var/log/httpd/mod_jk.shm.17823 childs=9
[Mon Jun 24 14:38:24 2019] [17877:139891516274752] [debug] wc_shutdown::jk_worker.c (390): Shutting down worker main_ajp13_worker

Looking at the AJPv3 documentation we see an explanation of the 8k chunks:

The container asks for more data from the request (If the body was too large to fit in the first packet sent over or when the request is chuncked). The server will send a body packet back with an amount of data which is the minimum of the request_length, the maximum send body size (8186 (8 Kbytes - 6)), and the number of bytes actually left to send from the request body. If there is no more data in the body (i.e. the servlet container is trying to read past the end of the body), the server will send back an "empty" packet, which is a body packet with a payload length of 0. (0x12,0x34,0x00,0x00)

In the above logs, I counted the number of times the 8k buffer was re-filled (36,856 times), and found that 301,703,216 bytes (287 MB, or 62%) of the file was transferred before the error occurred. On repeated attempts, this percentage varies wildly, but is usually over 300MB, and sometimes completes all 440MB successfully.

My interpretation of the above logs

The SSL handshake is succeeding, and the error occurs after the HTTP 100 Continue, so I suspect that the file has been partially sent prior to the following messages in mod_jk.log:

receiving data from client failed. Connection aborted or network problems
sending request to tomcat failed (unrecoverable), because of client read error (attempt=1)

Since there are no errors in catalina.out, and mod_jk is complaining about a client read error, I believe this rules out Tomcat itself (and the web application, Artifactory).

Most of the comments I see with a Google search on the above mod_jk messages state that they are transient network errors, but I'm not buying that since these are file transfers between to EC2 instances in the same region. I am also interpreting this as not having anything to do with a connection timeout, as that would (I think) produce a "Connection timed out after n milliseconds" message.

Changing the mod_jk and Tomcat configurations

Additional things I tried (these changes are cumulative, with no observable change in results):

upgraded mod_jk to the latest version (1.2.46)
increased worker packet size from 8,196 (default) to 16,384
increased worker packet size from 16,384 to 32,768
increased JkShmSize from 1kB (default) to 1GB (oops, value is specified in kB)
decreased JkShmSize from 1GB to 1MB
returned worker JkShmSize to default
returned worker packet size to default
increased worker retries from 2 (default) to 5
decreased reply_timeout from forever (default) to 1000ms (this actually caused some issues with the underlying service during successful uploads)
returned worker retries to default
returned reply_timeout to default
changed socket_keepalive from false (default) to true
returned socket_keepalive to default
changed Connector protocol from org.apache.coyote.http11.Http11NioProtocol to org.apache.coyote.http11.Http11Nio2Protocol
increased Tomcat connectionTimeout from 20 seconds (20000) to 60 seconds (60000)
set Tomcat connectionUploadTimeout to 60 seconds (60000)
set Tomcat disableUploadTimeout to false
returned connectionUploadTimeout and disableUploadTimeout to defaults
Forced chunked transfer via -H "Transfer-Encoding:chunked"
Set worker properties to: retries=2,socket_timeout=100,reply_timeout=30000
Reverted worker properties to defaults.
Added +DisableReuse to JkOptions.
Reverted JkOptions to defaults.
Created separate workers for the repo and repo-access virtual hosts.
Removed the non-SSL vhost entries, leaving only the two SSL virtual hosts.

Note: Separating the workers at least segregated the worker messages, so I know now that the error is occurring entirely on the repo web service, and not the repo-access web service:

[Sat Jun 29 16:16:55 2019] [6174:139690136819776] [info] ajp_read_into_msg_buff::jk_ajp_common.c (1550): (repo) receiving data from client failed. Connection aborted or network problems
[Sat Jun 29 16:16:55 2019] [6174:139690136819776] [info] ajp_process_callback::jk_ajp_common.c (2102): (repo) Reading from client aborted or client network problems
[Sat Jun 29 16:16:55 2019] [6174:139690136819776] [info] ajp_service::jk_ajp_common.c (2775): (repo) sending request to tomcat failed (unrecoverable), because of client read error (attempt=1)
[Sat Jun 29 16:16:55 2019] [6174:139690136819776] [info] jk_handler::mod_jk.c (2984): Aborting connection for worker=repo

Changing the HTTP(S) Client

I was able to reproduce the problem using wget instead of curl, which means that this isn't some esoteric issue with the client software:

-bash-4.2$ wget -v --tries=1 --user=backup --password=PASSWORD --header=X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3 --header=X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10 --method=PUT --body-file=snapshot-20190612.zip https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip

--2019-06-24 19:59:16--  https://repo.mydomain.org/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
Resolving repo.mydomain.org (repo.mydomain.org)... 123.222.100.132
Connecting to repo.mydomain.org (repo.mydomain.org)|123.222.100.132|:443... connected.
HTTP request sent, awaiting response... 401 401
Authentication selected: Basic realm="Artifactory Realm"
Reusing existing connection to repo.mydomain.org:443.
Giving up.

Testing on localhost

I've tested uploading from localhost on the archive server (https://localhost:443/), which requires adding the -k flag to curl in order to skip certificate hostname validation, and the upload succeeds every time (11 seconds). However, adding --limit-rate 20M causes the upload to fail every time (it works with --limit-rate 25M).

Bypassing Apache HTTPD, mod_jk and SSL works

Uploads always succeed if I bypass Apache HTTPD, mod_jk and SSL and go directly through port 8080:

[ec2-user@repo ~]$ curl -v -k --limit-rate 1024K -u backup:PASSWORD -H X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3 -H X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10 -T snapshot-20190612.zip http://localhost:8080/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
* Server auth using Basic with user 'backup'
> PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1
> Host: localhost:8080
> Authorization: Basic DELETED_TOKEN
> User-Agent: curl/7.61.1
> Accept: */*
> X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3
> X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10
> Content-Length: 496278603
> Expect: 100-continue
>
< HTTP/1.1 100
* We are completely uploaded and fine
< HTTP/1.1 201
< Server: Artifactory/5.3.2
...

Bypassing SSL only fails

Going through Apache HTTPD + mod_jk without SSL, I am still able to replicate the problem. This means that the problem has nothing to do with SSL.

[ec2-user@repo ~]$ curl -v -k --limit-rate 1024K -u backup:PASSWORD -H X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3 -H X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10 -T snapshot-20190612.zip http://localhost/resource-snapshot/snapshot/20190612/snapshot-20190612.zip
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
* Server auth using Basic with user 'backup'
> PUT /resource-snapshot/snapshot/20190612/snapshot-20190612.zip HTTP/1.1
> Host: localhost:8080
> Authorization: Basic DELETED_TOKEN
> User-Agent: curl/7.61.1
> Accept: */*
> X-Checksum-Sha1:6b9525bf9bb572ec7f70d0d474cb6533e0091bd3
> X-Checksum-Md5:6da0c444ad1b53dc1cf06784e204cb10
> Content-Length: 496278603
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Bypassing HTTPD and mod_jk

I tried bypassing Apache HTTPD and mod_jk by uploading directly to Tomcat via port 8443 (still using SSL). These requests succeeded every time. This leads me to believe that the problem is with Apache HTTPD and/or mod_jk, and is not caused by "transient network errors" or a firewall.

TL;DR

The upload failures only occur when going through HTTPD + mod_jk, not when communicating with Tomcat directly. SSL has been eliminated as a potential source of the problem. This is not a network problem because the error is reproducible when uploading from localhost. The issue appears to be related only to the amount of time it takes to transfer the file, not the size of the file.

My Question

Are there any configuration options that I should look into for HTTPD and/or mod_jk, workers.properties or even Tomcat to eliminate these problems?

Alternatively, is there any way to tell mod_jk to attempt to handle these failed buffer re-fills in a fault tolerant way, so that the uploads can successfully complete?

Parker

Asked: 2015-07-08 18:41:54 +0800 CST

A large number of JetDirect firmware updates - what issues do they address?

2

HP recently released a large number of firmware updates for JetDirect devices. There is no changelog provided. Curiously, many obsolete (e.g., 802.11b) devices are in the list, and the last time a firmware update was provided for some of these was 2004. The devices range from 10/100 ethernet to 802.11b/g print servers, EIO, MIO, and LIO. Does this batch of updates address some set of security vulnerabilities (e.g., SSLv3, POODLE or other documented CVEs) or introduce new functionality (e.g., TLS or WPA2)?

Update Based on information in the page supplied in the accepted answer, there were several security updates and new features introduced in these firmware updates. Among those for the 280m, these were the updates I was most interested in, as they appear to several SSL vulnerabilities and add WPA compatibility:

Automatic IP Addressing
Ethernet Locally Administered Addressing (LAA)
Import/Export of Certificates for SSL
CERT Vulnerability Note VU#412115 has been fixed
FTP and DHCP improvements, SPX and TCP improvements.
Wi-Fi Certification for 802.11b and WPA for the 680n (J6058A), 380x (J6061A), 280m (J6044A) 802.11b wireless print servers.

How to verify Certificate Revocation List(s) against multiple certification paths

Errors when attempting to connect to PostgreSQL 9.6 using SSL wildcard server certificate and no client certificates

Sporadic mod_jk Client Errors when uploading large files

A large number of JetDirect firmware updates - what issues do they address?

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?