Hello,
I have an ELK AIO installation that works just fine without TLS.
When I configure TLS for HTTP doesn't start and we have no indication why.
The certificate is signed from our internal CA which is configured on the server and the rest of the network.
Attached is the elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node1.example.com
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
## Security Related Configuration
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: "/etc/pki/elasticsearch/elk_pkey.pem"
xpack.security.http.ssl.certificate: "/etc/pki/elasticsearch/elk_cert.pem"
xpack.security.http.ssl.certificate_authorities: ['/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt']
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: ['_local_', '10.10.0.21']
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
## THE FOLLOWING LINE MUST BE REMOVED IF WE CREATE A CLUSTER
discovery.type: single-node
## THE PREVIOUS LINE MUST BE REMOVED IF WE CREATE A CLUSTER
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
/var/log/messages:
Jan 31 15:47:07 node1 systemd: Starting Elasticsearch...
Jan 31 15:47:09 node1 elasticsearch: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jan 31 15:47:12 node1 systemd: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jan 31 15:47:12 node1 systemd: Failed to start Elasticsearch.
Jan 31 15:47:12 node1 systemd: Unit elasticsearch.service entered failed state.
Jan 31 15:47:12 node1 systemd: elasticsearch.service failed.
Journal:
Jan 31 09:52:07 node1 systemd[1]: Starting Elasticsearch...
Jan 31 09:52:09 node1 elasticsearch[41328]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jan 31 09:52:17 node1 elasticsearch[41328]: ERROR: [1] bootstrap checks failed
Jan 31 09:52:17 node1 elasticsearch[41328]: [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must
Jan 31 09:52:17 node1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n
From /var/log/elasticsearch/elasticsearch.log:
[2020-01-31T16:07:02,381][ERROR][o.e.b.Bootstrap ] [node1.example.com] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
...
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - access to read certificate_authorities file [/etc/pki/elasticsearch/ca-bundle.trust.crt] is blocked; SSL resources should be placed in the [/etc/elasticsearch] directory
...
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/elasticsearch/ca-bundle.trust.crt" "read")
....
[2020-01-31T16:07:02,391][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1.example.com] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl]
....
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - access to read certificate_authorities file [/etc/pki/elasticsearch/ca-bundle.trust.crt] is blocked; SSL resources should be placed in the [/etc/elasticsearch] directory
....
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/elasticsearch/ca-bundle.trust.crt" "read")
....
And the certificates and the bundle:
ls -lrt /etc/pki/elasticsearch/
total 268
-rw-------. 1 elasticsearch elasticsearch 1675 Ιαν 31 14:11 elk_pkey.pem
-rw-r--r--. 1 elasticsearch elasticsearch 35 Ιαν 31 14:11 elk_cert.pem
-r--r--r--. 1 elasticsearch elasticsearch 263289 Ιαν 31 16:02 ca-bundle.trust.crt
Any help would be appreciated