Unable to retrieve version information from Elasticsearch nodes

Hi!
I trying to install EFK stack into my k8s GCP cluster.
I use official helm charts from https://helm.elastic.co: elasticsearch and kibana
I use automatically generated certificates and credentials on deployment startup.
I tried to use custom certs generated by Issuer, the manual is here
Here is my Issuer and Certificate config (I don't use it now):

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: elasticsearch
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: elasticsearch-master-certs
  namespace: elasticsearch
spec:
  isCA: true
  dnsNames:
    - elasticsearch-master
    - elasticsearch-master:9200
    - elasticsearch-master.elasticsearch
    - elasticsearch-master.elasticsearch:9200
    - elasticsearch-master.elasticsearch.svc
    - elasticsearch-master.elasticsearch.svc:9200
    - elasticsearch-master.elasticsearch.svc.cluster.local
    - elasticsearch-master.elasticsearch.svc.cluster.local:9200
  issuerRef:
    kind: Issuer
    name: selfsigned-issuer
  secretName: elasticsearch-master-certs
  subject:
    organizations:
      - org

Here is my values files:

elasticsearch:
  
  # createCert: false

  esConfig:
    elasticsearch.yml: |
      xpack.security.enabled: true
      xpack.security.transport.ssl.enabled: true
      xpack.security.http.ssl.enabled: true
      xpack.security.transport.ssl.verification_mode: "certificate"
      xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
      xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
      xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca.crt"
      xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
      xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
      xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca.crt"
  
  # secretMounts:
  #  - name: elasticsearch-master-certs
  #    secretName: elasticsearch-master-certs
  #    path: /usr/share/elasticsearch/config/certs
  #    defaultMode: 0755

  volumeClaimTemplate:
    accessModes: ["ReadWriteOnce"]
    resources:
      requests:
        storage: 2Ti

  resources:
    requests:
      cpu: "1000m"
      memory: "8Gi"
    limits:
      cpu: "1000m"
      memory: "16Gi"
  
  service:
    enabled: true
    annotations:
      cloud.google.com/load-balancer-type: "Internal"
    type: LoadBalancer
    loadBalancerIP: ***

  affinity:
    nodeAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
            - key: cloud.google.com/gke-nodepool
              operator: In
              values:
              - elasticsearch-master
  tolerations:
    - key: app
      operator: Equal
      value: elasticsearch
      effect: NoSchedule
kibana:
  elasticsearchHosts: https://elasticsearch-master.elasticsearch.svc:9200
  
  kibanaConfig:
    kibana.yml: |
      server.publicBaseUrl: https://kibana.***.**

  secretMounts:
   - name: elasticsearch-master-certs
     secretName: elasticsearch-master-certs
     path: /usr/share/elasticsearch/config/certs
     defaultMode: 0755
  
  tolerations:
    - key: app
      operator: Equal
      value: elasticsearch
      effect: NoSchedule

  ingress:
    enabled: true
    className: "nginx"
    pathtype: ImplementationSpecific
    annotations:
      cert-manager.io/cluster-issuer: cf-issuer
    hosts:
      - host: kibana.***.**
        paths:
          - path: /
    tls:
     - secretName: kibana-tls
       hosts:
         - kibana.***.**

when I open URLs I get:


Seems like all stuff works well but Kibana says in log this:
Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate
And shows that it is not ready.

Also I noticed, that elastic generates new certificates every couple minutes (may be every 5-10-15 mins). It is problem for me because I installed elastic and kibana to different namespaces. I don't know is it expected and normal behavior?

I believe I mentioned all. Appreciate your help.

it is damn magic
all fixed itself while I was writing this message.
my configs the same, don't know what it was.

now I get this:

Kibana says this

Hey @Dmitriy_Esin ,

Probably something wrong with certificates configuration, I suggest checking docs and this post has a similar issue Kibana not connecting to Elasticsearch when providing external Certificate

Regards, Dima

Hi!
Thank you for your answer!
I fixed it tonight :sweat_smile: and forgot to close the topic
I changed back configuration to self signed certs and tuned xpack.security configs
Now all works well!
thank you!

If someone need manifests:

elasticsearch:

  roles:
    - master
    - data
    - ingest
    - transform

  createCert: false

  esConfig:
    elasticsearch.yml: |
      xpack.security.enabled: true
      path.data: /usr/share/elasticsearch/data
      xpack.security.transport.ssl.enabled: true
      xpack.security.transport.ssl.verification_mode: "certificate"
      xpack.security.transport.ssl.client_authentication: required
      xpack.security.http.ssl.enabled: true
      xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
      xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
      xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca.crt"
      xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
      xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
      xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca.crt"
  
  secretMounts:
   - name: elasticsearch-master-certs
     secretName: elasticsearch-master-certs
     path: /usr/share/elasticsearch/config/certs
     defaultMode: 0755

  volumeClaimTemplate:
    accessModes: ["ReadWriteOnce"]
    resources:
      requests:
        storage: 2Ti

  resources:
    requests:
      cpu: "1000m"
      memory: "8Gi"
    limits:
      cpu: "1000m"
      memory: "16Gi"
  
  service:
    enabled: true
    annotations:
      cloud.google.com/load-balancer-type: "Internal"
    type: LoadBalancer
    loadBalancerIP: ***

  affinity:
    nodeAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
            - key: cloud.google.com/gke-nodepool
              operator: In
              values:
              - elasticsearch-master
  tolerations:
    - key: app
      operator: Equal
      value: elasticsearch
      effect: NoSchedule

kibana:
  elasticsearchHosts: https://elasticsearch-master.elasticsearch.svc:9200
  elasticsearch.ssl.certificateAuthorities: ["/usr/share/elasticsearch/config/certs/ca.crt"]
  server.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
  server.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
  server.ssl.enabled: true

  kibanaConfig:
    kibana.yml: |
      server.publicBaseUrl: https://kibana.***.**

  secretMounts:
   - name: elasticsearch-master-certs
     secretName: elasticsearch-master-certs
     path: /usr/share/elasticsearch/config/certs
     defaultMode: 0755
  
  tolerations:
    - key: app
      operator: Equal
      value: elasticsearch
      effect: NoSchedule

  ingress:
    enabled: true
    className: "nginx"
    pathtype: ImplementationSpecific
    annotations:
      cert-manager.io/cluster-issuer: cf-issuer
    hosts:
      - host: kibana.***.**
        paths:
          - path: /
    tls:
     - secretName: kibana-tls
       hosts:
         - kibana.***.**

Maybe you have any thoughts about this
I try to configure snapshots to google-storage
I use secret as keystore with json creds:

  keystore:
    - secretName: elasticsearch-keystore

But elastic says:
Cannot mount elasticsearch keystore in pod, device busy

ok, one problem less :slightly_smiling_face:

I can only point to the helm charts issue, that might give you some ideas to check.

Maybe change the topic headline to better describe the issue, it might help to get more folks from community.

Regards, Dima

thanks!
will try to fix :slight_smile: