Unable to load SSL configuration for Elasticsearch

Elastic Stack Elasticsearch
1

Hi, I am trying to Encrypt communications in Elasticsearch between nodes.

I refer to this, [Encrypting communications in Elasticsearch] (Encrypting communications in Elasticsearch | Elasticsearch Reference [7.11] | Elastic)

But I encountered such errors when I try to restart my elasticsearch. Below are the error logs stated in my elasticsearch log file:

[2021-03-23T19:00:08,214][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:529) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:525) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:143) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:458) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:290) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$16(Node.java:560) ~[elasticsearch-7.11.2.jar:7.11.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:564) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:75) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:438) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:527) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:525) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:143) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:458) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:290) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$16(Node.java:560) ~[elasticsearch-7.11.2.jar:7.11.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:564) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
... 6 more
Caused by: java.io.IOException: keystore password was incorrect
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2103) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:220) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1472) ~[?:?]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:98) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:438) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:527) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:525) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:143) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:458) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:290) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$16(Node.java:560) ~[elasticsearch-7.11.2.jar:7.11.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:564) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
... 6 more
[root@node-1 ~]#

Also my elasticsearch.yml configurations:

# Security
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
transport.tcp.compress: true

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.keystore.secure_password: "mypassword"
xpack.security.transport.ssl.truststore.secure_password: "mypassword"

# This turns on SSL for HTTP (Rest) interface
xpack.security.http.ssl.enabled: true

#This configures keystore to use for SSL on HTTP
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.http.ssl.truststore.path: certs/http.p12
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.client_authentication: optional

Can someone please help me on how can I fix this? I'm not sure where I did wrong, I tried to troubleshoot but I keep getting the same errors.

2

You cannot set secure_password in the YAML, you need to add it to the Elasticsearch keystore.

Did you apply a password when you created this file? The error is complaining that the password for this file is incorrect.

3

Does that mean I should remove these parameter settings from the elastic yaml file?

I also have added the secure password settings to the Elasticsearch keystore as you suggest, but I'm not sure if I did it correctly. Could you help take a look at the snippet below and verify it?

    root@node-3 bin]# /usr/share/elasticsearch/bin/elasticsearch-keystore list
    Enter password for the elasticsearch keystore :
    keystore.seed
    xpack.security.http.ssl.keystore.secure_password
    xpack.security.http.ssl.truststore.secure_password
    xpack.security.transport.ssl.keystore.secure_password
    xpack.security.transport.ssl.truststore.secure_password
    [root@node-3 bin]#
4

I comment out these parameters settings in the Elasticsearch YAML file

xpack.security.transport.ssl.keystore.secure_password: "mypassword"
xpack.security.transport.ssl.truststore.secure_password: "mypassword"

Then I restart the elasticsearch service, but I got another error below which from what I see it "failed to read keystore password on console". What could be the cause of this?

-- Unit elasticsearch.service has begun starting up.
Mar 24 13:56:43 node-3 kernel: xfs filesystem being remounted at /tmp supports timestamps until 2038 (0x7fffffff)
Mar 24 13:56:43 node-3 kernel: xfs filesystem being remounted at /var/tmp supports timestamps until 2038 (0x7fffffff)
Mar 24 13:56:44 node-3 systemd-entrypoint[7368]: Failed to read keystore password on console
Mar 24 13:56:44 node-3 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Mar 24 13:56:44 node-3 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
5

It looks like you have a password on the elasticsearch keystore.

If that is the case you need to provide that password to systemd, see: Starting Elasticsearch | Elasticsearch Guide [7.11] | Elastic

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.