# ElasticSearch renew certificate errors

**URL:** <https://discuss.elastic.co/t/elasticsearch-renew-certificate-errors/359452>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security, docker\
**Created:** [May 14, 2024, 11:17am UTC](https://discuss.elastic.co/t/elasticsearch-renew-certificate-errors/359452 "2024-05-14T11:17:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Boodle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boodle/32/86046_2.png) [@Boodle](https://discuss.elastic.co/u/Boodle)\
**Post date:** [May 14, 2024, 11:17am UTC](https://discuss.elastic.co/t/elasticsearch-renew-certificate-errors/359452/1 "2024-05-14T11:17:29Z")

</div>

Our SSL self signed elasticsearch certificates setup on docker had expired. Now Client application(Oracle Webcenter) is unable to make connection with Elasticsearch 7.17.6. Also trying to access elasticsearch cluster health via browser (https://:/\_cluster/health ) do not authorise the credentials and keeps on asking for them over again.

It is a 3 node cluster elasticsearch setup via docker-compose in a swarm with 1 node on each server and node 1 is the parent node. The SSL certificates for the nodes are in `/usr/share/elasticsearch/config/certificates` . the path `/usr/share/elasticsearch/config` is mounted as a local docker volume on each node

Followed below steps to renew certificate:

1. Inside container, ran 'Elasticsearch-certutil ca Es01.p12' & copied certificates to the respective certs folder
2. followed step 1 on all Nodes
3. The webcenter server wants a single certfile for the cluster but doesn't accept the CA PEM so following to creatae a consolidated .p12 file

- Extract the private and public keys for node 1 by executing `openssl pkcs12 -in es01.p12 -out es01_key.pem -nodes -nocerts` then `openssl pkcs12 -in es01.p12 -out es01.pem -nodes -nokeys`
- Extract the public key for node 2 by executing `openssl pkcs12 -in es02.p12 -out es02.pem -nodes -nokeys`
- Extract the public key for node 3 by executing `openssl pkcs12 -in es03.p12 -out es03.pem -nodes -nokeys`
- Finally combine node 1's private key and all of the public keys into a single .p12 file by executing `openssl pkcs12 -export -in es01.pem -inkey es01_key.pem -certfile es02.pem -certfile es03.pem -out es.p12`

Still gets error in logs TrustStore erros in logs.  
few queries:

1. Has above steps renewed both the certs and ca and if we have placed the files on elasticsearch server at correct paths. How to cross-check?
2. Moreover, how to resolve the issue 🙂 **All members -Kindly assist.**

`Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]`

Also, noticed below error in docker logs file

```auto
[2024-05-13T22:58:28,147][DEBUG][o.e.x.s.a.RealmsAuthenticator] [es01] Authentication of [kibana_system] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=TERMINATE, user=null, message=failed to authenticate user [kibana_system], exception=null}]

```

```auto

        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-05-13T22:58:20,648][DEBUG][r.suppressed] [es01] path: /_nodes, params: {filter_path=nodes.*.version,nodes.*.http.publish_address,nodes.*.ip}
org.elasticsearch.ElasticsearchSecurityException: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[x-pack-core-7.17.6.jar:7.17.6]

```

@yquirion @ikakavas @TimV @

---

<div class="post-metadata">

**Author:** ![Boodle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boodle/32/86046_2.png) [@Boodle](https://discuss.elastic.co/u/Boodle)\
**Post date:** [May 14, 2024, 5:31pm UTC](https://discuss.elastic.co/t/elasticsearch-renew-certificate-errors/359452/2 "2024-05-14T17:31:33Z")

</div>

also see below error in docker node logs

```auto
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:103) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
        ... 30 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:103) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
        ... 30 more
[2024-05-14T17:22:35,359][WARN][o.e.t.TcpTransport] [es01] exception caught on transport layer [Netty4TcpChannel{localAddress=/, remoteAddress=, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) ~[?:?]

```

@warkolm

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [May 17, 2024, 5:57am UTC](https://discuss.elastic.co/t/elasticsearch-renew-certificate-errors/359452/3 "2024-05-17T05:57:03Z")

</div>

> [@Boodle](#):
>
> Our SSL self signed elasticsearch certificates setup on docker had expired. Now Client application(Oracle Webcenter) is unable to make connection with Elasticsearch 7.17.6. Also trying to access elasticsearch cluster health via browser (https://:/\_cluster/health ) do not authorise the credentials and keeps on asking for them over agai

It is unlikely that these issues are related.

We should treat them as separate topics.

> [@Boodle](#):
>
> - Inside container, ran 'Elasticsearch-certutil ca Es01.p12' & copied certificates to the respective certs folder
> - followed step 1 on all Nodes

Did you write that correctly?  
You should only run `elasticsearch-certutil ca` once, on a single node to generate a new CA for your cluster. You definitely should not have a CA per node.
