AccessDeniedException starting elasticsearch after installing X-Pack

[Justin_Wells]

Steps to get to problem on ubuntu 16.04, single node:

Installed Elasticsearch 6.2.1 via .deb package
Installed X-Pack, ec2-discovery and s3-repository modules
Set bootstrap password using:

printf {{ bootstrap_password }} | ./elasticsearch-keystore add bootstrap.password

Changed the 3 built-in account passwords via API
Created server certs with these commands:

./certutil ca --pass {{ ca_pass }} --out /etc/elasticsearch/certs/elastic-stack-ca.p12
./certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --ca-pass {{ ca_pass }} --pass {{ cert_pass }} --out /etc/elasticsearch/certs/elastic.p12

Set permissions on these files as:

-rw------- 1 elasticsearch elasticsearch 3611 Jun  4 19:07 elastic.p12
-rw------- 1 elasticsearch elasticsearch 2607 Jun  4 19:07 elastic-stack-ca.p12

Added passwords to keystore and trust store using these commands:

printf {{ cert_pass }} | ./elasticsearch-keystore add --stdin xpack.security.transport.ssl.keystore.secure_password
printf {{ cert_pass }} | ./elasticsearch-keystore add --stdin xpack.security.transport.ssl.truststore.secure_password

Configured Elasticsearch.yml with:

cluster.name: production
node.name: ip-10-227-96-51 
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
http.port: 9200
transport.tcp.port: 9300
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone

discovery.ec2.protocol: http
discovery.ec2.endpoint: ec2.us-east-2.amazonaws.com
discovery.ec2.availability_zones: ["us-east-2a","us-east-2b"]
discovery.ec2.node_cache_time: 120s
discovery.ec2.tag.es_cluster: hgw-elasticsearch
discovery.ec2.groups: elastic-search-instance-SG 
discovery.zen.hosts_provider: ec2
discovery.zen.join_timeout: 90s 
discovery.zen.ping_timeout: 30s

xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/elastic-stack-ca.p12

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic.p12 

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/elastic.p12 

Started elasticsearch service and received:

Receiving the following:

Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic-stack-ca.p12
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic-stack-ca.p12
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at java.nio.file.Files.readAttributes(Files.java:1737)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at java.nio.file.Files.walkFileTree(Files.java:2706)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:147)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:119)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.cli.Command.main(Command.java:90)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jun  4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: #011at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)

Thanks for any help.

[Justin_Wells]

I did find that the keystore command should have the --stdin flag set when piping values to it, but after making this change I am still receiving the same error. I'll update the original post.

[TimV (Tim Vernum)]

There's a few things you'll need to fix up.

Firstly, you should remove this line:

xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/elastic-stack-ca.p12

• You have specified both xpack.security.http.ssl.truststore.path and certificate_authorities but you should only have one of them.
• certificate_authorities needs PEM formatted certificates, not PKCS#12
• You never need to use the CA PKCS#12 file ( elastic-stack-ca.p12) in any configuration. That file contains the private keys for your CA and your Elasticsearch nodes really shouldn't have access to those.

Secondly, you've configured passwords for transport (xpack.security.transport.ssl.keystore.secure_password) but not for http. You need to add the http passwords to the elasticsearch-keystore as well.

Finally, we need to resolve this issue (this specific one will go away when you change your config, but if this doesn't work, neither will the other files):

Jun 4 19:08:59 ip-10-227-96-51 elasticsearch[20428]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic-stack-ca.p12

You ned to double check the permissions on that file and the /etc/elasticsearch/certs/ directory.

[Justin_Wells]

Thanks for the clarification Tim, these changes got me rolling. I had tried troubleshooting with wide open permissions on the .p12 files previously, but turns out their parent directory needs to have 770.