Our SSL self signed elasticsearch certificates setup on docker had expired. Now Client application(Oracle Webcenter) is unable to make connection with Elasticsearch 7.17.6. Also trying to access elasticsearch cluster health via browser (https://:/_cluster/health ) do not authorise the credentials and keeps on asking for them over again.
It is a 3 node cluster elasticsearch setup via docker-compose in a swarm with 1 node on each server and node 1 is the parent node. The SSL certificates for the nodes are in /usr/share/elasticsearch/config/certificates . the path /usr/share/elasticsearch/config is mounted as a local docker volume on each node
Followed below steps to renew certificate:
Inside container, ran 'Elasticsearch-certutil ca Es01.p12' & copied certificates to the respective certs folder
followed step 1 on all Nodes
The webcenter server wants a single certfile for the cluster but doesn't accept the CA PEM so following to creatae a consolidated .p12 file
Extract the private and public keys for node 1 by executing openssl pkcs12 -in es01.p12 -out es01_key.pem -nodes -nocerts then openssl pkcs12 -in es01.p12 -out es01.pem -nodes -nokeys
Extract the public key for node 2 by executing openssl pkcs12 -in es02.p12 -out es02.pem -nodes -nokeys
Extract the public key for node 3 by executing openssl pkcs12 -in es03.p12 -out es03.pem -nodes -nokeys
Finally combine node 1's private key and all of the public keys into a single .p12 file by executing openssl pkcs12 -export -in es01.pem -inkey es01_key.pem -certfile es02.pem -certfile es03.pem -out es.p12
Still gets error in logs TrustStore erros in logs.
few queries:
Has above steps renewed both the certs and ca and if we have placed the files on elasticsearch server at correct paths. How to cross-check?
Moreover, how to resolve the issue All members -Kindly assist.
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
Also, noticed below error in docker logs file
[2024-05-13T22:58:28,147][DEBUG][o.e.x.s.a.RealmsAuthenticator] [es01] Authentication of [kibana_system] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=TERMINATE, user=null, message=failed to authenticate user [kibana_system], exception=null}]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-05-13T22:58:20,648][DEBUG][r.suppressed ] [es01] path: /_nodes, params: {filter_path=nodes.*.version,nodes.*.http.publish_address,nodes.*.ip}
org.elasticsearch.ElasticsearchSecurityException: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[x-pack-core-7.17.6.jar:7.17.6]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:103) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
... 30 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:103) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
... 30 more
[2024-05-14T17:22:35,359][WARN ][o.e.t.TcpTransport ] [es01] exception caught on transport layer [Netty4TcpChannel{localAddress=/, remoteAddress=, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) ~[?:?]
Did you write that correctly?
You should only run elasticsearch-certutil ca once, on a single node to generate a new CA for your cluster. You definitely should not have a CA per node.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.