ElasticSearch pods are in intializing dstate after configuring xpack security and tls

[s19k15 (Stavros Koureas)]

I am having an Elasticsearch deployment on a kubernetes cluster that was deployed with a 7.x chart and i changed the image to 8.x. This worked but now i need to enable new security feature, I have the following values:

- name: cluster.initial_master_nodes
  value: elasticsearch-master-0,
- name: discovery.seed_hosts
  value: elasticsearch-master-headless
- name: cluster.name
  value: elasticsearch
- name: network.host
  value: 0.0.0.0
- name: cluster.deprecation_indexing.enabled
  value: 'false'
- name: node.roles
  value: data,ingest,master,ml,remote_cluster_client

The Elasticsearch and kibana pods are able to start but i am unable to set APM Integration due security. So I am enabling security using the below values:

- name: xpack.security.enabled
  value: 'true'

Then i am getting an error log from the elasic search pod: "Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]". So i am enabling ssl using the below values:

- name: xpack.security.transport.ssl.enabled
  value: 'true'

Then i am getting an error log from Elasticsearch pod: "invalid SSL configuration for xpack.security.transport.ssl - server ssl configuration requires a key and certificate, but these have not been configured; you must set either [xpack.security.transport.ssl.keystore.path] (p12 file), or both [xpack.security.transport.ssl.key] (pem file) and [xpack.security.transport.ssl.certificate] (pem key file)".

I start with Option1, i am creating the keys using the below commands (no password / enter, enter / enter, enter, enter) and i am coping them to a persistent folder:

./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
cp elastic-stack-ca.p12 data/elastic-stack-ca.p12
cp elastic-certificates.p12 data/elastic-certificates.p12

In addition I am also configuring the below values:

- name: xpack.security.transport.ssl.truststore.path
  value: '/usr/share/elasticsearch/data/elastic-certificates.p12'
- name: xpack.security.transport.ssl.keystore.path
  value: '/usr/share/elasticsearch/data/elastic-certificates.p12'

But the pod is still in initializing, if generate the certificates with password. then i am getting an error log from Elasticsearch pod: "cannot read configured [PKCS12] keystore (as a truststore) [/usr/share/Elasticsearch/data/elastic-certificates.p12] - this is usually caused by an incorrect password; (no password was provided)"

Then i go to Option2, i am creating the keys using the below commands and i am coping them to a persistent folder

./bin/elasticsearch-certutil ca --pem
unzip elastic-stack-ca.zip –d

cp ca.crt data/ca.crt
cp ca.key data/ca.key

In addition I am also configuring the below values:

- name: xpack.security.transport.ssl.key
  value: '/usr/share/elasticsearch/data/ca.key'
- name: xpack.security.transport.ssl.certificate
  value: '/usr/share/elasticsearch/data/ca.crt'

But the pod is still in initializing state without providing any logs.

At last i located the same issue to the eleastic search community, without any response: Elasticsearch pods are not ready when xpack.security.enabled is configured

Any ideas?