Tomcat [9.0.26] - SSLHandshake Exception - spring-mvc

Question

My application in tomcat (version 9.0.26) is interacting with a third party HTTPS webservice. In the proces of SSL negotiation, the handshake fails and I am looking for help with the debug.
Tomcat startup parameters are:
INFO: Command line argument:
-Djavax.net.ssl.trustStore=C:\tomcat32\9.0.26\conf\MyTrustStore.p12 Dec 08, 2019 8:56:08 AM
org.apache.catalina.startup.VersionLoggerListener log INFO: Command
line argument: -Djavax.net.ssl.trustStorePassword=MyPass Dec 08, 2019
8:56:08 AM org.apache.catalina.startup.VersionLoggerListener log INFO:
Command line argument: -Djavax.net.ssl.trustStoreType=PKCS12 Dec 08,
2019 8:56:08 AM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument:
-Djavax.net.debug=ssl:handshake:verbose:keymanager:trustmanager Dec 08, 2019 8:56:08 AM org.apache.catalina.core.AprLifecycleListener
lifecycleEvent INFO: Loaded APR based Apache Tomcat Native library
[1.2.23] using APR version [1.7.0]. Dec 08, 2019 8:56:08 AM
org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true]. Dec 08, 2019 8:56:08 AM
org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO:
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
Dec 08, 2019 8:56:08 AM org.apache.catalina.core.AprLifecycleListener
initializeSSL INFO: OpenSSL successfully initialized [OpenSSL 1.1.1c
28 May 2019] Dec 08, 2019 8:56:09 AM
org.apache.coyote.AbstractProtocol init INFO: Initializing
ProtocolHandler ["http-nio-8080"] Dec 08, 2019 8:56:10 AM
org.apache.coyote.AbstractProtocol init INFO: Initializing
ProtocolHandler ["ajp-nio-8009"] Dec 08, 2019 8:56:10 AM
org.apache.catalina.startup.Catalina load INFO: Server initialization
in [2,592] milliseconds Dec 08, 2019 8:56:10 AM
org.apache.catalina.core.StandardService startInternal INFO: Starting
service [Catalina]
On enabling SSL debug logs, I captured below on the logs
Allow unsafe renegotiation: false Allow legacy hello messages: true Is
initial handshake: true Is secure renegotiation: false Ignoring
unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for
TLSv1 Ignoring unsupported cipher suite:
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1 Ignoring unsupported
cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1 Ignoring
unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for
TLSv1 Ignoring unsupported cipher suite:
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1 Ignoring unsupported
cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1 Ignoring
unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for
TLSv1 Ignoring unsupported cipher suite:
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1 Ignoring
unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for
TLSv1.1 Ignoring unsupported cipher suite:
TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1 Ignoring unsupported
cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite:
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1 Ignoring unsupported
cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1 Ignoring
unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for
TLSv1.1 %% No cached client session update handshake state:
client_hello1 upcoming handshake states: server_hello[2]
* ClientHello, TLSv1.2 RandomCookie: GMT: 1558998647 bytes = { 181, 223, 221, 91, 197, 4, 57, 190, 202, 50, 65, 37, 54, 151, 211, 23, 88,
35, 181, 111, 187, 68, 160, 166, 229, 25, 76, 123 } Session ID: {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
TLS_RSA_WITH_AES_256_CBC_SHA256,
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA,
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA,
TLS_DHE_DSS_WITH_AES_256_CBC_SHA,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
TLS_RSA_WITH_AES_128_CBC_SHA256,
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA,
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA,
TLS_DHE_DSS_WITH_AES_128_CBC_SHA,
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_RSA_WITH_AES_256_GCM_SHA384,
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1,
secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1,
sect571r1, secp256k1} Extension ec_point_formats, formats:
[uncompressed] Extension signature_algorithms, signature_algorithms:
SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA,
SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA1withECDSA,
SHA1withRSA, SHA1withDSA Extension extended_master_secret Extension
server_name, server_name: [type=host_name (0),
value=certservicesgateway.Bingonline.com]
http-nio-8080-exec-3, WRITE: TLSv1.2 Handshake, length = 236 http-nio-8080-exec-3, READ: TLSv1.2 Handshake, length = 89 check
handshake state: server_hello[2]
ServerHello, TLSv1.2 RandomCookie: GMT: 1119462208 bytes = { 96, 236, 134, 31, 185, 89, 247, 95, 189, 217, 105, 127, 42, 183, 115, 120,
142, 31, 103, 111, 54, 50, 166, 58, 130, 107, 63, 128 } Session ID:
{15, 155, 163, 64, 244, 187, 119, 250, 40, 154, 103, 47, 201, 208,
211, 136, 114, 116, 248, 159, 173, 34, 212, 74, 194, 65, 71, 17, 39,
181, 196, 228} Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Compression Method: 0 Extension renegotiation_info,
renegotiated_connection: Extension ec_point_formats, formats:
[uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
* %% Initialized: [Session-6, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
** TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
And towards the end of the handshake failure find the 'Invalidated' logger:
update handshake state: change_cipher_spec upcoming
handshake states: client finished[20] upcoming handshake states:
server change_cipher_spec[-1] upcoming handshake states: server
finished[20] http-nio-8080-exec-3, WRITE: TLSv1.2 Change Cipher Spec,
length = 1
* Finished verify_data: { 124, 94, 237, 141, 218, 48, 210, 88, 98, 142, 112, 197 }
* update handshake state: finished[20] upcoming handshake states: server change_cipher_spec[-1] upcoming handshake states: server
finished[20] http-nio-8080-exec-3, WRITE: TLSv1.2 Handshake, length =
40 http-nio-8080-exec-3, READ: TLSv1.2 Alert, length = 2
http-nio-8080-exec-3, RECV TLSv1.2 ALERT: fatal, handshake_failure %%
Invalidated: [Session-6, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
http-nio-8080-exec-3, called closeSocket() http-nio-8080-exec-3,
handling exception: javax.net.ssl.SSLHandshakeException: Received
fatal alert: handshake_failure 2019-12-07 23:00:43.732 INFO ---
[nio-8080-exec-3] .v.w.t.MyServiceHandler : ######### Other Exception
happened in
MyServiceHandler.execute():com.sun.xml.internal.ws.client.ClientTransportException:
HTTP transport error: javax.net.ssl.SSLHandshakeException: Received
fatal alert: handshake_failure, and the cause
is:javax.net.ssl.SSLHandshakeException: Received fatal alert:
handshake_failure
com.sun.xml.internal.ws.client.ClientTransportException: HTTP
transport error: javax.net.ssl.SSLHandshakeException: Received fatal
alert: handshake_failure at
com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(Unknown
Source) at
com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown
Source) at
com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown
Source) at
com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown
Source) at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown
Source) at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown
Source) at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown
Source) at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown
Source)
From the loggers, I think the SSL protocol version used is TLS1.2 and that looks good. It looks like the Cipher is not matching - but unsure if I am overlooking something and how to remediate this.
On inspecting the logs further, found this one error:
Unparseable certificate extensions: 1 1: ObjectId: 2.5.29.31
Criticality=false Unparseable CRLDistributionPoints extension due to
java.io.IOException: invalid URI
name:ldap://Enroll.visaca.com:389/cn=Visa Information Delivery
External CA,c=US,ou=Visa International Service
Association,o=VISA?certificateRevocationList
Update - 09-Dec -- In consultation with our middleware support team who indicate that the above CRL exception could be a false alarm.
So with that assumption, looking closely at the log and comparing with the steps outlined in TLS handshake steps wiki.
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
http-nio-8080-exec-15, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished verify_data: { 27, 249, 167, 252, 151, 220, 110, 252, 113, 134, 248, 228 }
*** update handshake state: finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
http-nio-8080-exec-15, WRITE: TLSv1.2 Handshake, length = 40
http-nio-8080-exec-15, READ: TLSv1.2 Alert, length = 2
http-nio-8080-exec-15, RECV TLSv1.2 ALERT: fatal, handshake_failure
%% Invalidated: [Session-6, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
http-nio-8080-exec-15, called closeSocket() http-nio-8080-exec-15,
handling exception: javax.net.ssl.SSLHandshakeException: Received
fatal alert: handshake_failure
The step 'server change_cipher_spec' is where I am suspecting something has gone wrong - though unsure how to debug this further. Appreciate any pointers.

Answer 1

Finally the issue is resolved.
As expected, 'Unparseable certificate extensions' turned out to be a false alarm.
Finally setting both trust store and key store helped resolve. The key store was also require for client authentication during the SSL Handshake.
-Djavax.net.ssl.trustStore=C:\Users\cert\visatomcat.p12 -Djavax.net.ssl.trustStorePassword=pass123 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.keyStore=C:\Users\cert\visatomcat.p12 -Djavax.net.ssl.keyStorePassword=pass123