Error: [o.e.b.Elasticsearch ] fatal exception while booting Elasticsearchorg.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchSecurityException:

Elastic Stack Elasticsearch
1

Hi All,

I am running into an ssl certificate issue when trying to form a cluster with 2 Elasticsearch nodes created on 2 AWS EC2 servers spread across 2 subnets. I followed the steps in the below ref link to generate CA and http file to enable SSL connection between the 2 nodes. elastic-stack-ca.p12 and http.p12 files we copied to the respective locations. I not sure if there is something that I am missing here, any help is much appreciated.

Below are my ES yaml files and logs shared from both instances named node-1 and node-2

node-1-elasticsearch-yml

cluster.name: demo-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_local_,_site_]
http.port: 9200
discovery.seed_hosts: ["10.2.58.202", "10.2.62.26"]
discovery.seed_providers: ec2
discovery.ec2.endpoint: ec2.us-west-2.amazonaws.com
discovery.ec2.tag.cluster_name: demo-cluster
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
logger.org.elasticsearch.discovery.ec2: "TRACE"
cluster.initial_master_nodes: ["node-1", "node-2"]
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
  truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
http.host: 0.0.0.0

node-2-elasticsearch.yml

cluster.name: demo-cluster
node.name: node-2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_local_,_site_]
http.port: 9200
discovery.seed_hosts: ["10.2.58.202", "10.2.62.26"]
discovery.seed_providers: ec2
discovery.ec2.endpoint: ec2.us-west-2.amazonaws.com
discovery.ec2.tag.cluster_name: demo-cluster
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
logger.org.elasticsearch.discovery.ec2: "TRACE"
cluster.initial_master_nodes: ["node-1", "node-2"]
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
  truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
http.host: 0.0.0.0

node-1-cluster-logs

[2022-07-01T17:49:10,841][ERROR][o.e.b.Elasticsearch      ] [node-1] fatal exception while booting Elasticsearch
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (a keystore password was provided)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:228) [elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) [elasticsearch-8.3.1.jar:?]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (akeystore password was provided)
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:605) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (a keystore password was provided)
        at org.elasticsearch.common.ssl.SslFileUtil.ioException(SslFileUtil.java:56) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:98) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.lambda$createTrustManager$0(CompositeTrustConfig.java:48) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.createTrustManager(CompositeTrustConfig.java:51) ~[?:?]
       at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more
Caused by: java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2158) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:226) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1503) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.lambda$createTrustManager$0(CompositeTrustConfig.java:48) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.createTrustManager(CompositeTrustConfig.java:51) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2158) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:226) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1503) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.lambda$createTrustManager$0(CompositeTrustConfig.java:48) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.createTrustManager(CompositeTrustConfig.java:51) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more

node-2-cluster-logs

2022-07-01T17:49:38,128][ERROR][o.e.b.Elasticsearch      ] [node-2] fatal exception while booting Elasticsearch
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (a keystore password was provided)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:228) [elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) [elasticsearch-8.3.1.jar:?]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (akeystore password was provided)
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:605) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (a keystore password was provided)
        at org.elasticsearch.common.ssl.SslFileUtil.ioException(SslFileUtil.java:56) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:98) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.lambda$createTrustManager$0(CompositeTrustConfig.java:48) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.createTrustManager(CompositeTrustConfig.java:51) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
       at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more
Caused by: java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2158) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:226) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1503) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.lambda$createTrustManager$0(CompositeTrustConfig.java:48) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.createTrustManager(CompositeTrustConfig.java:51) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2158) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:226) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1503) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.lambda$createTrustManager$0(CompositeTrustConfig.java:48) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.common.ssl.CompositeTrustConfig.createTrustManager(CompositeTrustConfig.java:51) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:310) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:668) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:235) ~[elasticsearch-8.3.1.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:681) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.1.jar:?]
        ... 1 more

Reference link: Set up basic security for the Elastic Stack | Elasticsearch Guide [8.0] | Elastic

Note: Additionally, I installed ec2 plugins, setup jvm options and memlock.

2

The error message is pretty telling :

You have set xpack.security.http.ssl.keystore.secure_password in elastisearch secure settings but you have added the wrong value.

You can see the current value with

./bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password

To verify that it’s wrong , and then set it again with

./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
3

After following the above suggestion, I was able to form the cluster successfully. However, we are seeing an intermittent response delay from the ES cluster(~20s delay in fetching results). Here are some logs, I have reviewed a few questions in the forum for the similar issues but the solution presented in those cases do not seem to apply to our situation.


[2022-07-15T20:29:26,863][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/10.146.X.X:9200, remoteAddress=/10.104.Y.Y:65485}
[2022-07-15T20:29:26,875][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/10.146.X.X:9200, remoteAddress=/10.104.Y.Y:65488}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.