Elasticsearch enable security issues

[willelastz]

Hi,

I have a Elasticsearch 7.6 cluster installed base on

https://github.com/openstack/openstack-helm-infra/tree/master/elasticsearch

Following is what I did to enable security:
a. Generate certificate

    ./bin/elasticsearch-certutil ca
    File location: /usr/share/elasticsearch/elastic-stack-ca.p12

    ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
    File location: /usr/share/elasticsearch/elastic-certificates.p12

    kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12

b. Enable Security on statefulset for master pod
kubectl edit statefulset elasticsearch-master

     ----
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.security.transport.ssl.enabled
          value: "true"
        - name: xpack.security.transport.ssl.verification_mode
          value: certificate
        - name: xpack.security.transport.ssl.keystore.path
          value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
        - name: xpack.security.transport.ssl.truststore.path
          value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    ----
        - mountPath: /usr/share/elasticsearch/config/certs
          name: elastic-certificates
          readOnly: true
    ----
      - name: elastic-certificates
        secret:
          defaultMode: 444
          secretName: elastic-certificates

c. Enable security on statefulset for data pod
kubectl edit statefulset elasticsearch-data

    ----
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.security.transport.ssl.enabled
          value: "true"
        - name: xpack.security.transport.ssl.verification_mode
          value: certificate
    ----
        - mountPath: /usr/share/elasticsearch/config/certs
          name: elastic-certificates
    ----
      - name: elastic-certificates
        secret:
          defaultMode: 444
          secretName: elastic-certificates

d. Enable security on deployment for client
kubectl edit deployment elasticsearch-client

    ----
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.security.transport.ssl.enabled
          value: "true"
        - name: xpack.security.transport.ssl.verification_mode
          value: certificate
        - name: xpack.security.transport.ssl.keystore.path
          value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
        - name: xpack.security.transport.ssl.truststore.path
          value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    ----
        - mountPath: /usr/share/elasticsearch/config/certs
          name: elastic-certificates
    ----
      - name: elastic-certificates
        secret:
          defaultMode: 444
          secretName: elastic-certificates

After pods restart, I got the following issue:
a. data pots are stuck in init stage

        kubectl get pod |grep data
        elasticsearch-data-0                                  1/1     Running     0          42m
        elasticsearch-data-1                                  0/1     Init:0/3    0          10m
        kubectl logs elasticsearch-data-1 -c init |tail -1
Entrypoint WARNING: <date/time> entrypoint.go:72: Resolving dependency Service elasticsearch-logging in namespace osh-infra failed: Service elasticsearch-logging has no endpoints .

b. Client pod errors regarding connection refused

    Warning  Unhealthy  18m (x4 over 19m)     kubelet, s1-worker-2  Readiness probe failed: Get http://192.180.71.82:9200/_cluster/health: dial tcp 192.180.71.82:9200: connect: connection refused
    Warning  Unhealthy  4m17s (x86 over 18m)  kubelet, s1-worker-2  Readiness probe failed: HTTP probe failed with statuscode: 401

c. Service "elasticsearch-logging" endpoints is empty

Any suggestions how to fix or what is wrong?
Thanks.