Persistent CertificateException error on running any of the elasticsearch tools in docker

bere_test · December 3, 2023, 6:44am

I have set up an Elasticsearch and Kibana stack with the docker-compose.yml file obtained from here - https://github.com/elastic/elasticsearch/blob/main/docs/reference/setup/install/docker/docker-compose.yml. I can run kibana without issues but when I try to run any of the elasticsearch tools, I keep getting the same error. Sample command run:

docker compose exec es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password --username logstash_system

Error observed:

WARNING: Owner of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [elasticsearch]
WARNING: Owner of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [elasticsearch]
04:59:07.836 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [172.22.0.3]; the server provided a certificate with subject name [CN=es01], fingerprint [9504284fd5fd18da7bb69577cf18bbac0b79b3e7], no keyUsage and no extendedKeyUsage; the certificate is valid between [2023-12-02T19:24:04Z] and [2026-12-01T19:24:04Z] (current time is [2023-12-03T04:59:07.827166700Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:localhost,IP:127.0.0.1,DNS:es01]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [2f26fe5098ff87b9a825801dfa6aa620f697a5ec] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/usr/share/elasticsearch/config/certs/ca/ca.crt})])
java.security.cert.CertificateException: No subject alternative names matching IP address 172.22.0.3 found
	at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:160) ~[?:?]
	at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:431) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
	at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:80) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1302) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:586) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141) ~[?:?]
	at org.elasticsearch.xpack.core.common.socket.SocketAccess.lambda$doPrivileged$0(SocketAccess.java:42) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
	at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:41) ~[?:?]
	at org.elasticsearch.xpack.core.security.CommandLineHttpClient.execute(CommandLineHttpClient.java:178) ~[?:?]
	at org.elasticsearch.xpack.core.security.CommandLineHttpClient.execute(CommandLineHttpClient.java:112) ~[?:?]
	at org.elasticsearch.xpack.security.tool.BaseRunAsSuperuserCommand.checkClusterHealthWithRetries(BaseRunAsSuperuserCommand.java:214) ~[?:?]
	at org.elasticsearch.xpack.security.tool.BaseRunAsSuperuserCommand.execute(BaseRunAsSuperuserCommand.java:127) ~[?:?]
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54) ~[elasticsearch-8.11.1.jar:8.11.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) ~[elasticsearch-cli-8.11.1.jar:8.11.1]
	at org.elasticsearch.cli.Command.main(Command.java:50) ~[elasticsearch-cli-8.11.1.jar:8.11.1]
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64) ~[cli-launcher-8.11.1.jar:8.11.1]

I tried setting the network.publish.host parameter to es.example.com for the 3 elasticsearch containers but that just gives me a different error

ERROR: Failed to determine the health of the cluster. , with exit code 69

I'm not sure how to proceed from here.

stephenb · December 3, 2023, 7:16am

I think you got a couple choices.

One you could exec into the container and then try again.

2nd

docker compose exec es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password --username logstash_system -u https://es01:9200

Or try localhost

If you look at the error message it says your certificate valid for

That's from the containers perspective

bere_test · December 3, 2023, 4:47pm

Thanks for the recommendations! None of them worked though.

I get the same error when I exec into the container and try again as well as when I run the command with the additional url flag (https://es01:9200, localhost, https://localhost:9200).

I'm now wondering if I need to generate new certs with the container ip addresses but not sure that's the right approach since that likely means I'll always need to do this if the containers ever need to be recreated.

stephenb · December 3, 2023, 6:48pm

OK I have repeatable getting same give me a bit to look... .

stephenb · December 3, 2023, 6:58pm

Running the official docker compose

And then execing into the es01 container

I got the exact same exception when I ran this command.

sh-5.0$ /usr/share/elasticsearch/bin/elasticsearch-reset-password --username logstash_system                            
18:55:48.131 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [172.21.0.3]; the server provided a certificate with subject name [CN=es01], fingerprint [0b3a5e62b016818c43d5f98db41211f57c9b41f2], no keyUsage and no extendedKeyUsage; the certificate is valid between [2023-12-03T18:45:52Z] and [2026-12-02T18:45:52Z] (current time is [2023-12-03T18:55:48.127856372Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:localhost,IP:127.0.0.1,DNS:es01]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [ad55e887ce1a9202968986fddde7ff93c4a7272f] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/usr/share/elasticsearch/config/certs/ca/ca.crt})])
java.security.cert.CertificateException: No subject alternative names matching IP address 172.21.0.3 found
        at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:160) ~[?:?]
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]

.......
ERROR: Failed to determine the health of the cluster. , with exit code 69

But then I ran this, it was fine / worked

sh-5.0$ /usr/share/elasticsearch/bin/elasticsearch-reset-password --username logstash_system -url https://127.0.0.1:9200
This tool will reset the password of the [logstash_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]n

This worked as well

h-5.0$ /usr/share/elasticsearch/bin/elasticsearch-reset-password --username logstash_system -url https://es01:9200
This tool will reset the password of the [logstash_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]

bere_test · December 3, 2023, 8:46pm

Thank you! That worked like a charm

system · December 31, 2023, 8:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.