XPack Security Enabled TLS/SSL configured, still not working!

S3l3ct3d · December 15, 2020, 1:38pm

Good morning, We are putting ELK through a POC and have configure and 5 node cluster in the MS Azure space. Currently we have 3 ES nodes, 1 Kibana node and 1 Logstash node. I have been trying to work on getting the xpack security configured for TLS/SSL, but I am running into errors due to access denied.

Below is the logs from /var/log/elasticsearch/

[2020-12-14T19:14:35,491][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [redacted] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.0.jar:7.9.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:526) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]
at org.elasticsearch.xpack.core.ssl.TrustConfig.unreadableTrustConfigFile(TrustConfig.java:121) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:70) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:437) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.0.jar:7.9.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:526) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:95) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:65) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:437) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.0.jar:7.9.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:526) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]

Here is my elasticsearch.yml file

cluster.name: "redacted"
node.name: "redacted"
path.logs: /var/log/elasticsearch
path.data: /datadisks/disk1/elasticsearch/data
discovery.seed_hosts: ["redacted","redacted","redacted"]
cluster.initial_master_nodes: ["redacted"]
node.master: true
node.data: false
transport.host: localhost
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0
node.max_local_storage_nodes: 1
node.attr.fault_domain: 1
node.attr.update_domain: 1
cluster.routing.allocation.awareness.attributes: fault_domain,update_domain
bootstrap.memory_lock: true
xpack.license.self_generated.type: trial
xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12

Here is my permission settings for /etc/elasticsearch/

root@redacted:/etc/elasticsearch# ls -ltr
total 52
-rw-rw---- 1 elastic elastic     0 Aug 11 21:44 users_roles
-rw-rw---- 1 elastic elastic     0 Aug 11 21:44 users
-rw-rw---- 1 elastic elastic   197 Aug 11 21:44 roles.yml
-rw-rw---- 1 elastic elastic   473 Aug 11 21:44 role_mapping.yml
-rw-rw---- 1 elastic elastic 17671 Aug 11 21:44 log4j2.properties
drwxr-s--- 2 elastic elastic  4096 Aug 11 21:44 jvm.options.d
-rw-rw---- 1 elastic elastic  2847 Aug 11 21:44 elasticsearch.yml.bak
-rw-rw---- 1 elastic elastic  2379 Dec 11 20:48 jvm.options
-rw-rw---- 1 root    elastic   351 Dec 15 02:26 elasticsearch.keystore
drwxrwx--- 2 elastic elastic  4096 Dec 15 02:28 certs
-rw-r--r-- 1 elastic elastic   976 Dec 15 13:07 elasticsearch.yml

Here are permission settings for the specific certs used

root@redacted:/etc/elasticsearch/certs# ls -ltr
total 8
-rw-rw---- 1 elastic elastic 2527 Dec 15 02:21 elastic-stack-ca.p12
-rw-rw---- 1 elastic elastic 3443 Dec 15 02:23 elastic-certificates.p12

I am unsure of what is happening, I cannot start elasticsearch after adding tls configurations.

aaron-nimocks · December 15, 2020, 2:20pm

The error is saying it can't access /etc/elasticsearch/elastic-certificates.p12 which doesn't exist since you have your certs in a /certs/ folder.

I'd try just moving your certs one folder down and see what it says. This shouldn't be the answer, just a test.

Either way I don't understand why elastic is looking for your certs in a different location than what is in your yml config.

S3l3ct3d · December 15, 2020, 2:30pm

@aaron-nimocks,

Thank you for the suggestion. I have moved the certs as you suggested, and I am still getting the same exact error. Which you can see below

[2020-12-14T19:14:35,491][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [redacted] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.0.jar:7.9.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:526) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]
        ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]
        at org.elasticsearch.xpack.core.ssl.TrustConfig.unreadableTrustConfigFile(TrustConfig.java:121) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:70) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:437) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.0.jar:7.9.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:526) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:95) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:65) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:437) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.0.jar:7.9.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:526) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]
        ... 6 more

Here is the updated elasticsearch yaml file

cluster.name: "redacted"
node.name: "redacted"
path.logs: /var/log/elasticsearch
path.data: /datadisks/disk1/elasticsearch/data
discovery.seed_hosts: ["redacted","redacted","redacted"]
cluster.initial_master_nodes: ["redacted"]
node.master: true
node.data: false
transport.host: localhost
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0
node.max_local_storage_nodes: 1
node.attr.fault_domain: 1
node.attr.update_domain: 1
cluster.routing.allocation.awareness.attributes: fault_domain,update_domain
bootstrap.memory_lock: true
xpack.license.self_generated.type: trial
xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

Here is the updated permissions for that location

root@redacted:/etc/elasticsearch# ls -ltr
total 60
-rw-rw---- 1 elastic elastic     0 Aug 11 21:44 users_roles
-rw-rw---- 1 elastic elastic     0 Aug 11 21:44 users
-rw-rw---- 1 elastic elastic   197 Aug 11 21:44 roles.yml
-rw-rw---- 1 elastic elastic   473 Aug 11 21:44 role_mapping.yml
-rw-rw---- 1 elastic elastic 17671 Aug 11 21:44 log4j2.properties
drwxr-s--- 2 elastic elastic  4096 Aug 11 21:44 jvm.options.d
-rw-rw---- 1 elastic elastic  2847 Aug 11 21:44 elasticsearch.yml.bak
-rw-rw---- 1 elastic elastic  2379 Dec 11 20:48 jvm.options
-rw-rw---- 1 elastic elastic  2527 Dec 15 02:21 elastic-stack-ca.p12
-rw-rw---- 1 elastic elastic  3443 Dec 15 02:23 elastic-certificates.p12
-rw-rw---- 1 root    elastic   351 Dec 15 02:26 elasticsearch.keystore
drwxrwx--- 2 elastic elastic  4096 Dec 15 14:22 certs
-rw-r--r-- 1 elastic elastic   964 Dec 15 14:24 elasticsearch.yml

aaron-nimocks · December 15, 2020, 3:02pm

Been awhile since I've done this but try adding config like the example in the docs. No full path. The documentation suggests it reads from the /config/certs/ folder.

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Also change your certs folder to /home/elasticsearch/config/certs/ and move your certs into there as per step 4.

TimV · December 16, 2020, 4:11am

This is coming from pretty low down in the JVM's file handling and means that your O/S is preventing the Elasticsearch process from reading that file.

To work out what's going on we need to start with some fundamentals about how you're running Elasticsearch:

How did you install Elasticsearch (zip/tar/docker/rpm/deb ?)
How are you starting Elasticsearch?
Which user does the Elasticsearch process run as?
At the beginning of your log file, after you try to start ES, there should be a line that says JVM arguments [...]. Can you post that here please?

system · January 13, 2021, 4:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.