Error When Trying To Setup Basic Security For The Elastic Stack

Hello Everyone,

I am currently trying to get basic security setup for the elastic stack using this tutorial: Set up basic security for the Elastic Stack | Elasticsearch Guide [7.13] | Elastic

I am using a Ubuntu virtual machine. I have already followed the minimal security tutorial right before this one.

The first thing I did was generate a CA for my cluster. I then generated a certificate and private key for my node. This is the point where the tutorial got a bit bumpy. The tutorial mentions that you should "Copy the elastic-certificates.p12 file to the ES_PATH_CONF directory on every node in your cluster." I am running a one node setup. I didn't quite understand what it mean when it said "ES_PATH_CONF" so I just put the certificates in the same directory as my elasticsearch.yml file. This directory is cd /etc/elasticsearch. I used the command cp elastic-certificates.p12 /etc/elasticsearch. When I did this command I checked my directory and the copy was successful. I then followed the instructions to configure the .yml file. My full .yml file can be seen below:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: false
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

discovery.type: single-node
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
network.host: 0.0.0.0
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: [127.0.0.1]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

I then ran these two commands and put the corret password in.

./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

When I try to restart Elasticsearch I get the following error

Starting elasticsearch (via systemctl): elasticsearch.serviceJob for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
 failed!

I then checked the log at /var/log/elasticsearch/my-application.log. The contents of the log can be found below:

[2021-07-12T16:24:06,579][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.2.jar:7.13.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.2.jar:7.13.2]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:454) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:298) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$18(Node.java:605) ~[elasticsearch-7.13.2.jar:7.13.2]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:609) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.2.jar:7.13.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.2.jar:7.13.2]
        ... 6 more

I have been troubleshooting this issue for quite some time and have a few ideas on what could be the problem.

  • I know the tutorial talks about the difference between the HTTP transport port and the Elasticsearch transport port. I am I supposed to make a new group of certificates? Do I need to specify this new group in the elasticsearch.yml file? Is the process for making this new group the same as making the first one?
  • Is it possible that there is something wrong with the first time I setup the minimal security. Maybe I typed the wrong password when I ran the command that can only be ran once (The interactive password setup).

Summary: I understand that the confusion is coming from a lack of understanding rather than a problem with the software but any help is still greatly appreciated.

Thanks,
Jared

The error message

org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];

means the elasticsearch process has no permission to read the certificate file /etc/elasticsearch/elastic-certificates.p12. If the directory /etc/elasticsearch/ is indeed where the elasticsearch.yml file lives, you just need to change the file permission of the certificate file. Specifically you need to ensure the two have the same permissions. Since you cp the certificate file manually into the directory, I assume you did it with either your own user or a root user. Both would result in a different set of file permissions to be set for the certificate file.

To fix the issue, first check the permission of the elasticsearch.yml file with ls -l /etc/elasticsearch/elasticsearch.yml. You should see something like the following:

-rw-r--r--   1 elasticsearch  elasticsearch  ...... elasticsearch.yml`

Note the user and group values from the above output (they both are elasticsearch). You can then fix the permission of the certificate with the chown command, e.g.:

sudo chown elasticsearch: /etc/elasticsearch/elastic-certificates.p12

Thank You! That worked perfectly!