Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate

I'm running a one node Elastic cluster with Elasticsearch and Kibana. The configuration worked before on an earlier version about a year ago, however, after Kibanas version automatically updated due to new container images, the following error started appearing:

kubectl logs kibana-pod

{"type":"log","@timestamp":"2022-02-21T08:47:48+00:00","tags":["error","Elasticsearch-service"],"pid":7,"message":"Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate"}

kubectl version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:24:08Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

I use Let'sEncrypt certificates for Elasticsearch and Kibana. I also renewed the certificates to make sure the configuration has fresh ones.

I use the helm charts for both Elasticsearch and Kibana, however, I use some custom values:

kibana_values.yaml

---
elasticsearchHosts: "https://redacted"

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: 'KIBANA_ENCRYPTION_KEY'
    valueFrom:
      secretKeyRef:
        name: kibana
        key: encryptionkey



secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/kibana/config/certs-gen/


kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs-gen/privkey2.pem
      certificate: /usr/share/kibana/config/certs-gen/cert2.pem
    xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs-gen/fullchain2.pem
      verificationMode: certificate

protocol: https

service:
  type: NodePort
  loadBalancerIP: ""
  port: 5601
  nodePort: 30002
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  httpPortName: http

These are the values for my elastic_search.yml

replicas: 1
minimumMasterNodes: 1

esConfig:
   elasticsearch.yml: |
     xpack.security.enabled: true
     xpack.security.transport.ssl.enabled: true
     xpack.security.transport.ssl.verification_mode: certificate
     xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs-gen/privkey2.pem
     xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs-gen/cert2.pem
     xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs-gen/fullchain2.pem" ]
     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.verification_mode: certificate
     xpack.security.http.ssl.key:  /usr/share/elasticsearch/config/certs-gen/privkey2.pem
     xpack.security.http.ssl.certificate:  /usr/share/elasticsearch/config/certs-gen/cert2.pem
     xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs-gen/fullchain2.pem" ]


extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs-gen/
protocol: https
service:
  labels: {}
  labelsHeadless: {}
  type: NodePort
  nodePort: 30001
  annotations: {}
  httpPortName: http
  transportPortName: transport
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""
  clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

The solution is to change:

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs-gen/privkey2.pem
      certificate: /usr/share/kibana/config/certs-gen/cert2.pem
    xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs-gen/fullchain2.pem
      verificationMode: certificate

to

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      keystore.path: /usr/share/kibana/config/certs-gen/keystore.pkcs12
      truststore.path: /usr/share/kibana/config/certs-gen/keystore.pkcs12
      keystore.password: ""
      truststore.password: ""
    xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}

and

esConfig:
   elasticsearch.yml: |
     xpack.security.enabled: true
     xpack.security.transport.ssl.enabled: true
     xpack.security.transport.ssl.verification_mode: certificate
     xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs-gen/privkey2.pem
     xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs-gen/cert2.pem
     xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs-gen/fullchain2.pem" ]
     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.verification_mode: certificate
     xpack.security.http.ssl.key:  /usr/share/elasticsearch/config/certs-gen/privkey2.pem
     xpack.security.http.ssl.certificate:  /usr/share/elasticsearch/config/certs-gen/cert2.pem
     xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs-gen/fullchain2.pem" ]

to

esConfig:
   elasticsearch.yml: |
     xpack.security.transport.ssl.enabled: true
     xpack.security.transport.ssl.verification_mode: certificate
     xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.enabled: true

both stores were generated as follows:

cat privkey2.pem > store.pem
cat cert2.pem >> store.pem
openssl pkcs12 -export -in store.pem -out keystore.pkcs12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.