Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate

[Behzad_Nazemi]

Dear Elastic Team,

Would you please help us with this issue? The new nodes could not join the cluster since we can not create a token for nodes. The http.p12 is already created and contained a PrivateKey but the error is still there. Either for genetrating tokens for Kibana or nodes.

Kind Regards,

[frank_2 (Frank)]

I also have this problem.

I have created a CA, and then created individual node .p12 files, and distributed them to the nodes. The certificates are working, as I can bootstrap a cluster and add masters to a total of 3.

When I come to generate a token to enrol a Kibana or node, I get the same error message as above.

[Behzad_Nazemi]

Hi Frank

Yes exactly same goes for me and the joining token did not generated. Did you find any other solutions for join new nodes to cluster? At the moment I have a single node cluster that I want to add two more nodes to it.

BR
Behzad

[frank_2 (Frank)]

Yes, I have given up on using the enrolment tokens as the process seems to depend on using an auto-generated CA and distributing the key for that CA to all nodes in the cluster, which seems somewhat insecure.

There is a Github issue which seems relevant: Generating enrolment token for Kibana should not require the CA key · Issue #89017 · elastic/elasticsearch · GitHub

Instead, I manually generated the certificates for my cluster, distributed them, and bootstrapped it. To get three master nodes online, one must configure the cluster.initial_master_nodes value in your elasticsearch.yml or environment.

Once you have those masters up, you can comment out that line in the config.

Adding non-master nodes is done by just pointing new nodes at the masters through discovery.seed_hosts. They then, slightly alarmingly, just join the cluster without any further authentication or authorisation.

It may be down to the fact I have enrolment enabled, or that the nodes are using signed, trusted certificates, but I'm pretty sure I didn't enable client authentication on the transport / http config.

I guess This Is Fine...?

[Behzad_Nazemi]

Thank you Frank for your kind attention and cooperation.
we have only three nodes and all of these should be master. I just revised the elasticsearch.yml and put the hostname of these three nodes in the cluster.initial_master_nodes, and my elasticsearch.yml for one of my nodes is like this

bootstrap.memory_lock: false
cluster.initial_master_nodes:
- elastic-01
- elastic-02
- elastic-03
cluster.name: my-cluster
cluster.routing.allocation.enable: all
network.bind_host: 0.0.0.0
network.host: 0.0.0.0
node.name: elastic-01
thread_pool.search.queue_size: 8000
thread_pool.search_throttled.queue_size: 1000
thread_pool.write.queue_size: 8000
transport.host : 0.0.0.0
http.port: 9200
xpack.security.audit.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: false

node.attr.box_type: hot
node.attr.data: hot

path.data: /opt/elasticsearch/data
path.logs: /opt/elasticsearch/logs
action.auto_create_index: true

but it seems each node is independent and did NOT join the cluster, although these nodes have access to each other. When I restarted the elastic service and get cluster health via curl it shows as follows

curl -XGET 'localhost:9200/_cluster/health?pretty' -u elastic:behzad
{
  "cluster_name" : "my-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

Frank, Could you please your elasticsearch.yml ?
Also, you said that you manually generated the certificate for the cluster, and on the other hand, you said that didn't enable client authentication on the transport / http config. Would you please describe it for me since it's a bit confusing?

Kind Regards,
Behzad