Hello Everyone,
I am currently trying to get basic security setup for the elastic stack using this tutorial: Set up basic security for the Elastic Stack | Elasticsearch Guide [7.13] | Elastic
I am using a Ubuntu virtual machine. I have already followed the minimal security tutorial right before this one.
The first thing I did was generate a CA for my cluster. I then generated a certificate and private key for my node. This is the point where the tutorial got a bit bumpy. The tutorial mentions that you should "Copy the elastic-certificates.p12 file to the ES_PATH_CONF directory on every node in your cluster." I am running a one node setup. I didn't quite understand what it mean when it said "ES_PATH_CONF" so I just put the certificates in the same directory as my elasticsearch.yml file. This directory is cd /etc/elasticsearch. I used the command cp elastic-certificates.p12 /etc/elasticsearch. When I did this command I checked my directory and the copy was successful. I then followed the instructions to configure the .yml file. My full .yml file can be seen below:
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
discovery.type: single-node
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
network.host: 0.0.0.0
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: [127.0.0.1]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
I then ran these two commands and put the corret password in.
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
When I try to restart Elasticsearch I get the following error
Starting elasticsearch (via systemctl): elasticsearch.serviceJob for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
failed!
I then checked the log at /var/log/elasticsearch/my-application.log. The contents of the log can be found below:
[2021-07-12T16:24:06,579][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.2.jar:7.13.2]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.2.jar:7.13.2]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:454) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:298) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:605) ~[elasticsearch-7.13.2.jar:7.13.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:609) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.2.jar:7.13.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.2.jar:7.13.2]
... 6 more
I have been troubleshooting this issue for quite some time and have a few ideas on what could be the problem.
- I know the tutorial talks about the difference between the HTTP transport port and the Elasticsearch transport port. I am I supposed to make a new group of certificates? Do I need to specify this new group in the elasticsearch.yml file? Is the process for making this new group the same as making the first one?
- Is it possible that there is something wrong with the first time I setup the minimal security. Maybe I typed the wrong password when I ran the command that can only be ran once (The interactive password setup).
Summary: I understand that the confusion is coming from a lack of understanding rather than a problem with the software but any help is still greatly appreciated.
Thanks,
Jared