Elasticsearch crashes with "Certificate chain is not valid" exception

[fetch (Ivan)]

Hello,

I have a cluster of 3 servers with ELK 6.8.6 stack.

I'm trying to setup the 3rd server of the cluster, but it behaves somehow different.

I specify the following configuration for ssl:

xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

xpack.security.http.ssl.enabled: false
#xpack.security.http.ssl.verification_mode: certificate
#xpack.security.http.ssl.key: my.key
#xpack.security.http.ssl.certificate: my.pem
#xpack.security.http.ssl.certificate_authorities: [ "my.ca" ]

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate

xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.keystore.password: "pass"

xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.password: "pass"

With this configuration elasticsearch starts and works fine. However, as soon as I uncomment xpack.security.http* options elasticsearch crashes at boot with the following traceback: https://pastebin.com/sKA7KUhg

It happens even with the options:

xpack.security.http.ssl.enabled: false
xpack.security.http.ssl.verification_mode: none

I've checked the certificate with the openssl and it seems fine:

openssl verify -verbose -CAfile my.ca my.pem
my.pem: OK

Another weird part of this situation that I have exactly the same my.key, my.pem, my.ca on two other servers and elasticsearch works fine with them.

Any ideas how to fix it?

[fetch (Ivan)]

3 days of trials and errors finally gave some results...

Two servers are using Oracle JDK that doesn't care about the pem with the certificates chain.

But the third server has OpenJDK and turns out that it will fail chain validation if the certificates placed in the wrong order. So the solution was to reorder the my.pem file in the way that the next certificate has the subject equal to the issuer of the pevious certificate.