Unable to get issuer certificate with TLS enabled Kibana & Elasticsearch in k8s

[Scottapotamus]

I am trying to enable TLS on Elasticsearch and kibana in kubernetes using the ECK operator. But when kibana tries to connect to Elasticsearch I get

{"type":"log","@timestamp":"**********","tags":["error","elasticsearch-servce"[],"pid":14,"message":"Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate"}

I am able to curl Elasticsearch, but have not been able to get any configuration working with kibana. I am using cert-manager to create my certificates.

kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  config:
    server.basePath: /kibana
    server.rewriteBasePath: false
    server.ssl.enabled: true
    server.ssl.key: /mnt/usr/tls.key
    server.ssl.certificate: /mnt/usr/tls.crt
    elasticsearch.ssl.certificateAuthorities: 
      - /mnt/usr/ca.crt
    elasticsearch.hosts:
      - https://default-es-http.elastic-cloud:9200
  count: 1
  http:
    service:
      metadata:
        annotations:
          konghq.com/protocol: https
    tls:
      certificate:
        secretName: kibana-certs-tls
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
        - name: certs
          mountPath: /mnt/usr
      volumes:
      - name: certs
        secret:
          secretName: kibana-certs-tls
  version: 7.17.0

default-Elasticsearch.yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: default
spec:
  nodeSets:
  - config:
      xpack.security.http.ssl.enabled: true
      xpack.security.http.ssl.client_authentication: optional
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    count: 1
    name: default
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
  http:
    service:
      spec:
        # expose this cluster Service with a ClusterIP and add public ingress for access
        type: ClusterIP
    tls:
      certificate:
        secretName: default-es-certs-tls
  version: 7.17.0

Any help would be appreciated. Thanks!

[Michael_Montgomery]

@Scottapotamus can we get more information around secret kibana-certs-tls? When I use cert-manager to create certificates for Elasticsearch, I get a secret with keys: tls.crt, and tls.key, but not ca.crt. Since the error message you're getting is saying that Kibana doesn't trust the CA that issued the ES certificate, it makes me think that the key ca.crt in the secret kibana-certs-tls is not the CA that created the certificates in the ES secret default-es-certs-tls. Hope this makes sense.