Elasticsearch pods are not ready when xpack.security.enabled is configured

[etrpchevska (Emilija)]

Hello, I am trying to enable minimal and then basic security on our Elasticsearch cluster following these guidelines:

Set up minimal security for Elasticsearch | Elasticsearch Guide [7.14] | Elastic

Set up basic security for the Elastic Stack | Elasticsearch Guide [7.14] |...

This is the Elasticsearch config file:

elasticsearch.yml: |
    cluster.name: dev-observability
    network.host: "0.0.0.0"
    discovery.zen.ping.unicast.hosts: es-cluster.dev-elastic-system.svc.cluster.local
    discovery.zen.minimum_master_nodes: 2
    cluster.initial_master_nodes: es-cluster-0, es-cluster-1, es-cluster-2
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 
    xpack.monitoring.enabled: true

I am trying to deploy 3 replicas with a k8s StatefulSet. When the xpack lines are commented out, the cluster is brought up successfully, but when I uncomment them, the first created pod never transitions into a ready state and thus the StatefulSet does not create the remaining two pods and the cluster is stuck in discovery phase and throws the following message:

{"type": "server", "timestamp": "2021-08-17T13:01:49,130Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "dev-observability", "node.name": "es-cluster-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-cluster-0, es-cluster-1, es-cluster-2] to bootstrap a cluster: have discovered [{es-cluster-0}{1lVbtXzkQLCFutz1CSplzQ}{KLwe2kTCQDaf8E7mpd-R8w}{10.84.8.11}{10.84.8.11:9300}{cdfhilmrstw}]; discovery will continue using [10.96.167.192:9300] from hosts providers and [{es-cluster-0}{1lVbtXzkQLCFutz1CSplzQ}{KLwe2kTCQDaf8E7mpd-R8w}{10.84.8.11}{10.84.8.11:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

Following the instructions in the pages I posted above, I generated an elastic-certificates.p12 file and mounted it as a ConfigMap to the pods in the following path:

/usr/share/elasticsearch/config/certs/elastic-certificates.p12

I also created a secret containing values for the elastic user and password and I mounted it as an environment variable in the StatefulSet:

- name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                name: elastic-credentials
                key: password
          - name: ELASTIC_USERNAME
            valueFrom:
              secretKeyRef:
                name: elastic-credentials
                key: username

As mentioned, when I disable the xpack security setting, the cluster is working as expected and the StatefulSet creates 3 pods which are able to discover each other. But, as soon as I enable the xpack settings, only 1 pod is created and it is stuck in the discovery phase.

What can I do to troubleshoot the issue further?