ElasticSearch renew certificate errors

Boodle · May 14, 2024, 11:17am

Our SSL self signed elasticsearch certificates setup on docker had expired. Now Client application(Oracle Webcenter) is unable to make connection with Elasticsearch 7.17.6. Also trying to access elasticsearch cluster health via browser (https://:/_cluster/health ) do not authorise the credentials and keeps on asking for them over again.

It is a 3 node cluster elasticsearch setup via docker-compose in a swarm with 1 node on each server and node 1 is the parent node. The SSL certificates for the nodes are in /usr/share/elasticsearch/config/certificates . the path /usr/share/elasticsearch/config is mounted as a local docker volume on each node

Followed below steps to renew certificate:

Inside container, ran 'Elasticsearch-certutil ca Es01.p12' & copied certificates to the respective certs folder
followed step 1 on all Nodes
The webcenter server wants a single certfile for the cluster but doesn't accept the CA PEM so following to creatae a consolidated .p12 file

Extract the private and public keys for node 1 by executing openssl pkcs12 -in es01.p12 -out es01_key.pem -nodes -nocerts then openssl pkcs12 -in es01.p12 -out es01.pem -nodes -nokeys
Extract the public key for node 2 by executing openssl pkcs12 -in es02.p12 -out es02.pem -nodes -nokeys
Extract the public key for node 3 by executing openssl pkcs12 -in es03.p12 -out es03.pem -nodes -nokeys
Finally combine node 1's private key and all of the public keys into a single .p12 file by executing openssl pkcs12 -export -in es01.pem -inkey es01_key.pem -certfile es02.pem -certfile es03.pem -out es.p12

Still gets error in logs TrustStore erros in logs.
few queries:

Has above steps renewed both the certs and ca and if we have placed the files on elasticsearch server at correct paths. How to cross-check?
Moreover, how to resolve the issue All members -Kindly assist.

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]

Also, noticed below error in docker logs file

[2024-05-13T22:58:28,147][DEBUG][o.e.x.s.a.RealmsAuthenticator] [es01] Authentication of [kibana_system] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=TERMINATE, user=null, message=failed to authenticate user [kibana_system], exception=null}]


        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-05-13T22:58:20,648][DEBUG][r.suppressed             ] [es01] path: /_nodes, params: {filter_path=nodes.*.version,nodes.*.http.publish_address,nodes.*.ip}
org.elasticsearch.ElasticsearchSecurityException: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[x-pack-core-7.17.6.jar:7.17.6]

@yquirion @ikakavas @TimV @

Boodle · May 14, 2024, 5:31pm

also see below error in docker node logs

        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:103) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
        ... 30 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:103) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
        ... 30 more
[2024-05-14T17:22:35,359][WARN ][o.e.t.TcpTransport       ] [es01] exception caught on transport layer [Netty4TcpChannel{localAddress=/, remoteAddress=, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) ~[?:?]

@warkolm

TimV · May 17, 2024, 5:57am

It is unlikely that these issues are related.

We should treat them as separate topics.

Did you write that correctly?
You should only run elasticsearch-certutil ca once, on a single node to generate a new CA for your cluster. You definitely should not have a CA per node.