Elasticsearch with TLS does not start

Hello,
I have an ELK AIO installation that works just fine without TLS.
When I configure TLS for HTTP doesn't start and we have no indication why.
The certificate is signed from our internal CA which is configured on the server and the rest of the network.

Attached is the elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node1.example.com
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch

## Security Related Configuration
xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: "/etc/pki/elasticsearch/elk_pkey.pem"
xpack.security.http.ssl.certificate: "/etc/pki/elasticsearch/elk_cert.pem"
xpack.security.http.ssl.certificate_authorities: ['/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt']

#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: ['_local_', '10.10.0.21']
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
## THE FOLLOWING LINE MUST BE REMOVED IF WE CREATE A CLUSTER
discovery.type: single-node
## THE PREVIOUS LINE MUST BE REMOVED IF WE CREATE A CLUSTER
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

/var/log/messages:

Jan 31 15:47:07 node1 systemd: Starting Elasticsearch...
Jan 31 15:47:09 node1 elasticsearch: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jan 31 15:47:12 node1 systemd: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jan 31 15:47:12 node1 systemd: Failed to start Elasticsearch.
Jan 31 15:47:12 node1 systemd: Unit elasticsearch.service entered failed state.
Jan 31 15:47:12 node1 systemd: elasticsearch.service failed.

Journal:

Jan 31 09:52:07 node1 systemd[1]: Starting Elasticsearch...
Jan 31 09:52:09 node1 elasticsearch[41328]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jan 31 09:52:17 node1 elasticsearch[41328]: ERROR: [1] bootstrap checks failed
Jan 31 09:52:17 node1 elasticsearch[41328]: [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must 
Jan 31 09:52:17 node1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n

From /var/log/elasticsearch/elasticsearch.log:

[2020-01-31T16:07:02,381][ERROR][o.e.b.Bootstrap          ] [node1.example.com] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
...
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - access to read certificate_authorities file [/etc/pki/elasticsearch/ca-bundle.trust.crt] is blocked; SSL resources should be placed in the [/etc/elasticsearch] directory
...
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/elasticsearch/ca-bundle.trust.crt" "read")

....
[2020-01-31T16:07:02,391][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1.example.com] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl]
....
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - access to read certificate_authorities file [/etc/pki/elasticsearch/ca-bundle.trust.crt] is blocked; SSL resources should be placed in the [/etc/elasticsearch] directory
....
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/elasticsearch/ca-bundle.trust.crt" "read")
....

And the certificates and the bundle:

ls -lrt /etc/pki/elasticsearch/
total 268
-rw-------. 1 elasticsearch elasticsearch   1675 Ιαν  31 14:11 elk_pkey.pem
-rw-r--r--. 1 elasticsearch elasticsearch     35 Ιαν  31 14:11 elk_cert.pem
-r--r--r--. 1 elasticsearch elasticsearch 263289 Ιαν  31 16:02 ca-bundle.trust.crt

Any help would be appreciated

I think the suggestion in the exception message is likely to help:

For security reasons Elasticsearch drops permissions to read files from directories outside its config path early on in its startup, so it cannot access /etc/pki/elasticsearch.

Yes. I found it about the same time you posted your answer, many thanks for this.

I moved the certificates in the /etc/elasticsearch/pki:

ls -lrt /etc/elasticsearch/pki
total 12
-rw-------. 1 elasticsearch elasticsearch 1675 Jan  31 16:21 elk_pkey.pem
-rw-r--r--. 1 root          elasticsearch   35 Jan  31 16:21 elk_cert.pem
-rw-r--r--. 1 root          elasticsearch 1814 Jan  31 16:31 ca-bundle.trust.crt

Now I get another issue:

Caused by: java.security.cert.CertificateException: No certificate data found

OK All fixed.
I had to use the CA's and not the sub-ordinate's CA certificate.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.