How to enable security on an existing 8.1 multi nodes docker cluster

jean.bissonnette · March 9, 2022, 11:09pm

I have an existing 5 nodes podman cluster. 4 nodes are on one VM, another node is on another VM.

Security is not enabled (xpack.security.enabled: false).

I tried to setup security by calling: bin/Elasticsearch-setup-passwords interactive
but it tells me that I have to set xpack.security.enabled to true

Well I changed xpack.security.enabled to true to all the nodes.
Result: the nodes fail to start.
There is a warning: it is trying to open a file under .aws. I cannot see this file since it is in the container. I am not running under AWS. What should I do?

{"type": "server", "timestamp": "2022-03-09T22:26:13,479Z", "level": "WARN", "component": "c.a.a.p.i.BasicProfileConfigFileLoader", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "Unable to load config file null",
"stacktrace": ["java.security.AccessControlException: access denied (\"java.io.FilePermission\" \"/usr/share/elasticsearch/.aws/config\" \"read\")",

Error message is:

ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/es-docker-cluster.log

Good, then I tried to enable xpack.security.transport.ssl.enabled and restart the cluster.
This time it tells me that I need a key.

{"type": "server", "timestamp": "2022-03-09T22:26:42,675Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchSecurityException: invalid SSL configuration for xpack.security.transport.ssl - server ssl configuration requires a key and certificate, but these have not been configured; you must set either [xpack.security.transport.ssl.keystore.path], or both [xpack.security.transport.ssl.key] and [xpack.security.transport.ssl.certificate]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-8.1.0.jar:8.1.0]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-8.1.0.jar:8.1.0]",
"Caused by: org.elasticsearch.ElasticsearchSecurityException: invalid SSL configuration for xpack.security.transport.ssl - server ssl configuration requires a key and certificate, but these have not been configured; you must set either [xpack.security.transport.ssl.keystore.path], or both [xpack.security.transport.ssl.key] and [xpack.security.transport.ssl.certificate]",
"at org.elasticsearch.xpack.core.ssl.SSLService.validateServerConfiguration(SSLService.java:635) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:612) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]",
"at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:474) ~[?:?]",
"at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:313) ~[?:?]",
"at org.elasticsearch.node.Node.lambda$new$16(Node.java:677) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.elasticsearch.node.Node.<init>(Node.java:691) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:284) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.1.0.jar:8.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-8.1.0.jar:8.1.0]",
"... 6 more"] }
uncaught exception in thread [main]
org.elasticsearch.ElasticsearchSecurityException: invalid SSL configuration for xpack.security.transport.ssl - server ssl configuration requires a key and certificate, but these have not been configured; you must set either [xpack.security.transport.ssl.keystore.path], or both [xpack.security.transport.ssl.key] and [xpack.security.transport.ssl.certificate]
        at org.elasticsearch.xpack.core.ssl.SSLService.validateServerConfiguration(SSLService.java:635)
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:612)
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156)
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:474)
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:313)
        at org.elasticsearch.node.Node.lambda$new$16(Node.java:677)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at org.elasticsearch.node.Node.<init>(Node.java:691)
        at org.elasticsearch.node.Node.<init>(Node.java:284)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
        at org.elasticsearch.cli.Command.main(Command.java:77)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/es-docker-cluster.log

How do I generate this key?
The nodes are not running, I do not have access to any Elasticsearch bin folder.
How do I generate a key out of a cluster that is not running?

There must be a way to enable security on an existing multi-node docker cluster.
Please help.

TimV · March 9, 2022, 11:45pm

Have you read the docs on this?

Manually configure security | Elasticsearch Guide [8.1] | Elastic

jean.bissonnette · March 10, 2022, 4:56pm

Sure I have read it.
But it is not clear to interpret the fundamental reasons of application behavior when running in a diff environment.
It is always a challenge to do it in a containerized world.
Mainly because if anything is wrong, the container fails and you do not have access to the bin folder to execute any command.
What I finally understood is that you have to start the containers WITHOUT any security (so they can run), then generate the certificates IN THE RUNNING CONTAINER, then copy the certificates to the config volumes on the host, then adjust the password on the keystore IN THE CONTAINER, then adjust Elasticsearch.yml to set xpack.security.transport.ssl values.
Then restart the container pod and services.

If the containers were not failing and just give warnings, it would be much easier and less confusing.

Nodes are now up with encrypted internode communication with SSL enabled.
Thank you

jean.bissonnette · March 16, 2022, 12:50pm

I followed carefully the instructions and the nodes are up.

I can verify by looking at logs (podman logf -f es01) or by curl to 9200

curl --cacert /var/elasticsearch/elasticsearch-config/es01/certs/kibana/elasticsearch-ca.pem -u elastic https://10.88.0.10:9200
Enter host password for user 'elastic':
{
  "name" : "es01",
  "cluster_name" : "es-docker-cluster",
  "cluster_uuid" : "doiewpWZT1CpyNuYRbK0Pw",
  "version" : {
    "number" : "8.1.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "3700f7679f7d95e36da0b43762189bab189bc53a",
    "build_date" : "2022-03-03T14:20:00.690422633Z",
    "build_snapshot" : false,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

The problem is to load Kibana.

Despite having followed the instructions found in the zip file generated while creating certificates for nodes and for Kibana, it still fails with this error.

[2022-03-16T12:36:39.935+00:00][ERROR][Elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_nodes?filter_path=nodes..version%2Cnodes..http.publish_address%2Cnodes.*.ip]

Here is my kibana.yml.

server.host: "10.88.0.10"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "https://10.88.0.10:9200"]
monitoring.ui.container.elasticsearch.enabled: false
xpack.reporting.enabled: false

elasticsearch.ssl.certificateAuthorities: [ "config/certs/kibana/elasticsearch-ca.pem" ]

Here is my Elasticsearch.yml

node.name: es01
cluster.name: es-docker-cluster
node.roles: master,data_hot,data_content,ingest,transform
bootstrap.memory_lock: true
network.host: 10.88.0.10
#172.16.82.115
transport.port: 9301

discovery.seed_hosts: ["10.88.0.10:9301","10.88.0.10:9302","10.89.0.10:9313"]
cluster.initial_master_nodes: ["es01","es02","es03"]

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
  client_authentication: optional

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

I have tried numerous ways to overcome in the last 5 days and was never able to succeed.
Please help.

TimV · March 17, 2022, 1:35am

Sorry, we don't really have a step-by-step guide to enabling security from scratch on an existing cluster in 8.x

We had something like that in 7.x, but there have been changes introduced in 8.0 that mean those docs are no longer relevant, and the replacements are a bit scattered.

The issue here is that Kibana is not configured to authenticate to Elasticsearch.

What you can do is exec into one of the Elasticsearch containers, and run:

bin/elasticsearch-reset-password -u kibana_system

That will generate a password for Kibana to use (you can also use the Elasticsearch change password API for this if you prefer).

Then modify your kibana.yml to add:

elasticsearch.username: kibana_system
elasticsearch.password: <<the new password>>

If you create a new cluster with security enabled, then all of this is done for you. However, when enabling security on an existing cluster some manual steps are required.

jean.bissonnette · March 17, 2022, 2:07pm

It worked, Thank you.

system · April 14, 2022, 2:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.