SSLHandshakeException when using High Level REST Client from Docker

Ewoks · June 9, 2019, 7:09pm

I am trying to access 6.x ES instance using High Level REST Client 6.7.2.
Access to this ES instance is provided to me via hostname (https://****.azureedge.net), username & password.

My Spring Boot application is getting data from the same ES without issues when it runs locally (from IDE), but throws SSLHandshakeException as soon as I try run it from Docker container (from my development machine or K8s cluster in cloud).

READ: TLSv1.2 alert, length = 2
javax.net.ssl|DEBUG|12|I/O dispatcher 1|null:-1|Received alert 
message (
    "Alert": {
        "level"      : "fatal",
        "description": "handshake_failure"
    }
)

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:938)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1749)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1708)
at org.elasticsearch.client.SecurityClient.getSslCertificates(SecurityClient.java:508) 
....
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:271)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:316)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)

Not sure if it is important but docker image is build from "debian:stretch-slim" with Java 11.
Do I need to add following to Dockerfile:

ADD my_certificate.crt:/container/cert/path
RUN update-ca-certificates

If so how to get my_certificate.crt, if I don't control ES instance? I assume I shouldn't add it, casue in my case I am just consuming https, no?!?

Any ideas how to resolve this? Thanks in advance

ikakavas · June 10, 2019, 4:57am

handshake_failure can be caused by a lot of things, from cipher/protocol incompatibility, to certificate trust issues. The best way forward would be to start your java application in the container with -Djavax.net.debug=ssl and look through the output to determine what the actual cause is.

By the way, since you mention the certificate yourself. What about the certificate of https://**** ? Is it signed by a trusted CA ? By a company/organization specific CA ? Is it self signed ? Have you done anything specific to make this work in your dev environment?

Ewoks · June 10, 2019, 8:11am

In the meantime I made some progress using -Djavax.net.debug=all and it turns out that in docker image happen just few first steps of usual ssl handshaking:

Produced ClientHello handshake message
WRITE: TLS13 handshake, length = 2352
Raw write
Raw read 
READ: TLSv1.2 alert, length = 2
Received alert message (
   "Alert": {
        "level"      : "fatal",
        "description": "handshake_failure"
    }
)

while running in local environment handshaking completes:

Produced ClientHello handshake message
WRITE: TLS13 handshake, length = 460
Raw write
Raw read
READ: TLSv1.2 handshake, length = 155
Consuming ServerHello
ServerHello
Negotiated protocol version: TLSv1.3
Session initialized:  Session(1560119025211|TLS_AES_256_GCM_SHA384)
WRITE: TLS13 change_cipher_spec, length = 1
Raw write
Raw read
READ: TLSv1.2 change_cipher_spec, length = 1
Consuming ChangeCipherSpec message
Raw read
READ: TLSv1.2 application_data, length = 27
...
Raw read
READ: TLSv1.2 application_data, length = 8469
Consuming server Certificate handshake message
... // here is the list of 3 certificates with "SHA256withRSA", "SHA256withRSA", "SHA1withRSA" signature algorithms
Found trusted certificate ⇢ SHA1withRSA
...

Related to your CA question:
"issuer" : "CN=Microsoft IT TLS CA 2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US", as well as "issuer" : "CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE", I suppose because ES runs in Azure.

I didn't do anything specific to make this work in my MacOS Java 11.0.2 dev env.

Ewoks · June 10, 2019, 11:41am

Changing Docker image from "slim" to not slim version, doesn't change anything.
Niether using Java 11.0.2 instead of first tried 11.0.1.

Update:
I ran curl with BasicAuth from inside Docker image, against same ES instance and that communication worked, meaning handshake completed without issues. I could execute small search query q=*:* which implies that docker image is OK.

That leaves us with HighLevel Rest Client issue, no?
Can it be that it behaves differently inside docker image and my dev environment?!

ikakavas · June 10, 2019, 1:58pm

curl uses /etc/ssl/certs in Debian, the JVM uses /JAVA_HOME/lib/security/cacerts as the source of trusted CAs.

Did you point it to the same https url ?
Can you share your client configuration ? Do you use mutual TLS Authentication by any chance ?

Ewoks · June 10, 2019, 2:28pm

I pointed to the exactly same https url

I create client as following:

  final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
  credentialsProvider.setCredentials(AuthScope.ANY,
          new UsernamePasswordCredentials(properties.getUsername(), properties.getPass()));

  return new RestHighLevelClient(
          RestClient
                  .builder(HttpHost.create(properties.getHost()))
                  .setHttpClientConfigCallback(
                          httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
                  )
  );

That's all related to authentication I am using. To me is provided just host, username & password.

OK, I learned something new about curl vs JVM today... If I understood correctly, there are no expected CAs in $JAVA_HOME/lib/security/cacerts for JVM, no? Can that be the reason for the issue?

TimV · June 11, 2019, 6:57am

Different JRE vendors supply different trusted certificates. In particular, for licensing reasons some OpenJDK builds ship with less certificates than Oracle builds, or the Operating System certificates.

It sounds like you might be running into that problem.

If that is indeed the problem, then your options are something like:

Ue a different JDK in your docker build
Symlink/copy the OS's trusted certs into the JRE
Manually add the certs you need.

Ewoks · June 11, 2019, 7:10am

Changing JDK now is pretty difficult to pull up
I already tried without success proposal using:

cd /etc/ssl/certs
for file in *.pem; do openssl x509 -outform der -in "$file" -out /tmp/certificate.der; keytool -import -alias "$file" -keystore ./java/cacerts -file /tmp/certificate.der -deststorepass changeit -noprompt; done;
how to get cert from that https://****.azureedge.net host? Btw. if host is in Azure Cloud shouldn't that be well known CA and it would surprising that OpenJDK doesn't deliver it out of the box?

P.S. I have other Microservice running in a Docker image built from almost the same dockerfile (same base image, same JDK, the only difference is which app jar to pack to the docker) that communicates with another host and has no issues out of the box. I understand different hosts need different certificates but it is just strange that certificates for Azure (it's not just some random 3rd party site) are missing, no?

TimV · June 11, 2019, 7:57am

OpenJDK 10 didn't have the AWS root cert in it, so nothing surprises me.
The issue was licensing, I can't recall the exact details, but I believe the issue was that the OpenJDK maintainers only wanted to ship CA files under an open source license.

I don't know whether this is the issue you have, but it seems plausible from what you've written.

Ewoks · June 11, 2019, 8:12am

Ok, roger that .

My next attempt is following:
As I could curl same host from inside docker image, I will try to add cert used by curl to $JAVA_HOME/lib/security/cacerts during building of docker image, all with hope that SpringBoot App will be able to use it

Ewoks · June 11, 2019, 9:46pm

Update:
got cert from host, added it during building of docker image to where JVM would look for it (sanity checked that after that docker image has +1 cert there) but still no overall success and same fck error again

Checked ciphers, seems that host expects ECDHE-RSA-AES256-GCM-SHA384, prepare docker image with "-Dcom.sun.net.ssl.enableECC=false", no positive results.

Interesting is ClientHello from local env and docker env is slightly different. "cipher suites" list is different but it is noticable that localy extension has no elements in the list, but docker ClientHello has extension with a lot of details like:

"supported_groups (10)": {"versions": [ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]},
signature_algorithms (13), 
signature_algorithms_cert (50), ..,
"status_request_v2 (17)": {"cert status request": {"certificate status type": ocsp_multi}} ..., "supported_versions (43)": {"versions": [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1]},

Ewoks · June 13, 2019, 11:26am

Final update:
For some reason, I needed to enforce TLSv1.2 on client, and handshake could complete after that.

Thanks everybody for support and ideas

system · July 11, 2019, 11:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.