I have configured JBOSS 5 with SSLVerifyClient="require"
- Client contains, server CA certs and there own certs(JDK 1.8).
- Server contains, client CA certs and there own certs(JDK 1.6).
For this case both CA's are different, when we are trying to communicate to that server and getting tlsv1 alert decrypt error
I have no clues what was the exception it is ?
During Curl,
Unknown SSL protocol error in connection
During OpenSSL:
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:error in SSLv3 flush data
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
The issue is probably (estimate based on OpenSSL output) that used algorithm mode is not supported by other side, or is considered deprecated/breached (i.e. TLSv1 is considered deprecated for more than a year).
I would recommend to capture handshake with tcpdump/wireshark. Since it seems that handshake is failing, you should be able to look into unencrypted handshake phase. Otherwise, since you have access to the private keys for the certs, you should be able to decrypt captured communication in wireshark too.
If supported-algorithm mismatch is not the issue, my second guess is DNS related. In some cases, it is not enough that client proved his identity with certificate, but there has to be a match among for example his DNS record (PTR) and CommonName field in the certificate.