Kibana unable to revive connection with Azure ECK

Hi all!

Having a bit of a struggle with getting Kibana talking to my elastic http service in ECK.

I'm running on an Azure cluster.

Kibana pod won't start, produces the following logs:

{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => certificate has expired"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-06-02T15:09:14Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-es-http.default.svc:9200/"}

The certificate is in fact not invalid. I know this because this self same certificate is installed on the elastic box. Config as follows:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.7.0
  http:
    service:
      spec:
        type: LoadBalancer
    tls:
      certificate:
        secretName: elasticsearch-es-cert
  nodeSets:
    - name: masters
      count: 3
      config: # Any setting that would go into your elasticsearch.yaml goes here
        node.master: true
        node.data: false
        node.ingest: false
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 30Gi
            storageClassName: standard
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command:
                ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
    - name: coord
      count: 1
      config: # Any setting that would go into your elasticsearch.yaml goes here
        node.master: false
        node.data: false
        node.ingest: false
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 30Gi
            storageClassName: standard
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command:
                ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
    - name: data
      count: 3
      config: # Any setting that would go into your elasticsearch.yaml goes here
        node.master: false
        node.data: true
        node.ingest: true
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 512Gi
            storageClassName: premium
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command:
                ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
          containers:
            - name: elasticsearch
              env:
                - name: ES_JAVA_OPTS
                  value: -Xms2g -Xmx2g
              resources:
                requests:
                  memory: 2Gi
                  cpu: 0.5
                limits:
                  memory: 4Gi
                  cpu: 1.5
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  podAffinityTerm:
                    labelSelector:
                      matchLabels:
                        elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
                    topologyKey: kubernetes.io/hostname
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: elasticsearch
spec:
  version: 7.7.0
  http:
    service:
      spec:
        type: LoadBalancer
      tls:
        certificate:
          secretName: elasticsearch-es-cert
  count: 1
  elasticsearchRef:
    name: elasticsearch
    namespace: default

I can connect to the elasticsearch cluster on it's loadbalancer's public IP address, SSL cert is valid.

If I remove the tls config from the elasticsearch cluster, Kibana fires right back up and we're off to the races.

Am I doing something wrong with how I'm configuring the tls on Elastic or Kibana?

Thanks!

What version of ECK are you running? There was a fix in 1.1.2 that allowed well known CAs to work automatically: https://www.elastic.co/guide/en/cloud-on-k8s/1.1/release-highlights-1.1.2.html

Hi Anya!

I'm using 1.1.2, I've only set up this cluster a day or two ago.

Node Kubernetes version:

I've followed the guide from the ECK docs: https://www.elastic.co/guide/en/cloud-on-k8s/1.1/k8s-deploy-elasticsearch.html

Do I need to make ECK aware of my cert in order for the Kibana and Elastic nodes to communicate?

Here's the describe of the kibana pod:

And from one of the elastic pods:

Name:           elasticsearch-es-masters-0
Namespace:      default
Priority:       0
Node:           aks-agentpool-14147237-vmss000001/10.240.0.5
Start Time:     Tue, 02 Jun 2020 14:12:15 +0200
Labels:         common.k8s.elastic.co/type=elasticsearch
                controller-revision-hash=elasticsearch-es-masters-57f568cd7c
                elasticsearch.k8s.elastic.co/cluster-name=elasticsearch
                elasticsearch.k8s.elastic.co/config-hash=31858377
                elasticsearch.k8s.elastic.co/http-scheme=https
                elasticsearch.k8s.elastic.co/node-data=false
                elasticsearch.k8s.elastic.co/node-ingest=false
                elasticsearch.k8s.elastic.co/node-master=true
                elasticsearch.k8s.elastic.co/node-ml=true
                elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-es-masters
                elasticsearch.k8s.elastic.co/version=7.7.0
                statefulset.kubernetes.io/pod-name=elasticsearch-es-masters-0
Annotations:    co.elastic.logs/module: elasticsearch
                update.k8s.elastic.co/timestamp: 2020-06-02T13:53:13.345656167Z
Status:         Running
IP:             [Redacted]
IPs:            <none>
Controlled By:  StatefulSet/elasticsearch-es-masters
Init Containers:
  elastic-internal-init-filesystem:
    Container ID:  docker://13c0c97db73b7368491136ca63e06ce692d4d8a003bf30eb5d4e364dbc7d63e0
    Image:         docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    Image ID:      docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b4ec018c974d23a796b1d17ddec3863e4e6deff76d25f58027c45b7a697089dc
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -c
      /mnt/elastic-internal/scripts/prepare-fs.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 02 Jun 2020 14:13:23 +0200
      Finished:     Tue, 02 Jun 2020 14:13:25 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_IP:     (v1:status.podIP)
      POD_NAME:  elasticsearch-es-masters-0 (v1:metadata.name)
      POD_IP:     (v1:status.podIP)
      POD_NAME:  elasticsearch-es-masters-0 (v1:metadata.name)
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-bin-local from elastic-internal-elasticsearch-bin-local (rw)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/elasticsearch-config-local from elastic-internal-elasticsearch-config-local (rw)
      /mnt/elastic-internal/elasticsearch-plugins-local from elastic-internal-elasticsearch-plugins-local (rw)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/transport-certificates from elastic-internal-transport-certificates (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
  sysctl:
    Container ID:  docker://28cc86d48a0f63975080bb0ea7b08eb5df7077dd257e93d94c51a94ed9be3294
    Image:         docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    Image ID:      docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b4ec018c974d23a796b1d17ddec3863e4e6deff76d25f58027c45b7a697089dc
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      sysctl -w vm.max_map_count=262144
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 02 Jun 2020 14:13:27 +0200
      Finished:     Tue, 02 Jun 2020 14:13:27 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      POD_IP:     (v1:status.podIP)
      POD_NAME:  elasticsearch-es-masters-0 (v1:metadata.name)
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
  elasticsearch:
    Container ID:   docker://351b1710ee0e278d7f35f43ac464c9728e797850c55769e7d49548fbce04a861
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    Image ID:       docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b4ec018c974d23a796b1d17ddec3863e4e6deff76d25f58027c45b7a697089dc
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 02 Jun 2020 14:13:28 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  2Gi
    Requests:
      memory:   2Gi
    Readiness:  exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
    Environment:
      POD_IP:                     (v1:status.podIP)
      POD_NAME:                  elasticsearch-es-masters-0 (v1:metadata.name)
      PROBE_PASSWORD_PATH:       /mnt/elastic-internal/probe-user/elastic-internal-probe
      PROBE_USERNAME:            elastic-internal-probe
      READINESS_PROBE_PROTOCOL:  https
      HEADLESS_SERVICE_NAME:     elasticsearch-es-masters
      NSS_SDB_USE_CACHE:         no
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  elasticsearch-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-data-elasticsearch-es-masters-0
    ReadOnly:   false
  downward-api:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
  elastic-internal-elasticsearch-bin-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  elastic-internal-elasticsearch-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-es-masters-es-config
    Optional:    false
  elastic-internal-elasticsearch-config-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  elastic-internal-elasticsearch-plugins-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  elastic-internal-http-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-es-http-certs-internal
    Optional:    false
  elastic-internal-probe-user:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-es-internal-users
    Optional:    false
  elastic-internal-remote-certificate-authorities:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-es-remote-ca
    Optional:    false
  elastic-internal-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      elasticsearch-es-scripts
    Optional:  false
  elastic-internal-transport-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-es-transport-certificates
    Optional:    false
  elastic-internal-unicast-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      elasticsearch-es-unicast-hosts
    Optional:  false
  elastic-internal-xpack-file-realm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-es-xpack-file-realm
    Optional:    false
  elasticsearch-logs:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

For Google's indexing for people also facing this problem, their search term would use the default cluster name and would be: "Unable to revive connection: https://quickstart-es-http.default.svc:9200/" just so any one else struggling with this issue can find this thread more easily.

I'm using a * cert. I think the issue is that Kibana is trying to use my cert when authenticating against Elasticsearch. Seeing as the internal address isn't using my external * cert DNS name, the cert verification is failing. How do I configure Kibana to use the internal ECK cert for the internal communication and the External cert for communication via the kubernetes service?

I'm not sure to understand your case. Do you have your own, private, certificate authority installed on the server ? Or are you deploying some certs issued by a known CA, like Let's Encrypt ?

Hi Michael!

My case is the following: I have an Azure Kubernetes hosted, ECK orchestrated Elastic search cluster. My understanding is that ECK sets up and manages the TLS authentication between the Elasticsearch and Kibana pods inside my Cluster.

I would like to be able to secure my Elasticsearch and Kibana services with SSL. I have a wildcard cert for my domain, *.domain.com issued by CommodoCA. This cert is working across various other services. I have a DNS a record that points elastic.mydomain.com to the loadbalancer for the elastic https service's kubernetes external IP address. Additionally I have a DNS a record that points kibana.mydomain.com to the loadbalancer for the Kibana https service's kubernetes external IP address.

I have uploaded the ca.crt, tls.crt and tls.key to a secret named elasticsearch-es-cert in my Kubernetes instance.

I have registered that under spec.http.tls.certificate.secretName in both my Elasticsearch and Kibana yaml files.

When connecting to elastic.mydomain.com:9200 I get SSL verification, confirming that my cert is working.

However, my Kibana pod will not start. When I check the logs, I get the errors mentioned in the initial post.

{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => certificate has expired"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-06-02T15:09:14Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-es-http.default.svc:9200/"}

More disconcertingly, when I remove the spec.http.tls.certificate.secretName on my elasticsearch pods, Kibana pods start right up, connect and run fine. When I visit kibana.mydomain.com I get SSL verification with the same cert.

I can only imagine that the internal https://elasticsearch-es-http.defauly.svc:9200 call for Kibana is trying to use my external cert, which will only validate on *.mydomain.com, and as such the SSL validation fails and Kibana can't connect and therefore start.

Have you or your teams seen anything like this before? Has anyone set up a secured ECK instance on Azure? It could be an issue around how the loadbalancers work? Maybe I'm missing something very obvious (I'm hoping this is the case).

Kibana should be configured by ECK with elasticsearch.ssl.verificationMode set to certificate, which means that hostname verification is skipped.

Could you exec a shell in the Kibana container and copy/paste the content of config/kibana.yml (remove the password or any sensitive url)

You should have something like this:

> kubectl exec -ti [Kibana Pod Name] /bin/bash
> cat config/kibana.yml
elasticsearch:
  hosts:
  - https://elasticsearch-es-http.default.svc:9200
  password: REDACTED
  ssl:
    verificationMode: certificate
....

Also the certificate has expired error is odd, could you execute a curl command from the Kibana Pod, it should print something like this:

> curl -k -v https://elasticsearch-es-http.default.svc:9200
[...]
* Server certificate:
* 	subject: CN=REDACTED
* 	start date: May 01 11:10:18 2020 GMT
* 	expire date: Jul 30 11:10:18 2020 GMT
* 	common name: REDACTED
* 	issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US

As mentioned before there is a known issue fixed in 1.1.2, could you also check that there is no error in the ECK logs ?

Hi Michael

The output of cat config/kibana.yml:

elasticsearch:
  hosts:
  - https://elasticsearch-es-http.default.svc:9200
  password: [Redacted]
  ssl:
    certificateAuthorities: /usr/share/kibana/config/elasticsearch-certs/ca.crt    
    verificationMode: certificate
  username: default-elasticsearch-kibana-user
server:
  host: "0"
  name: elasticsearch
  ssl:
    certificate: /mnt/elastic-internal/http-certs/tls.crt
    enabled: true
    key: /mnt/elastic-internal/http-certs/tls.key
xpack:
  license_management:
    ui:
      enabled: false
  monitoring:
    ui:
      container:
        elasticsearch:
          enabled: true
  security:
    encryptionKey: [Redacted]

This falls inline with what you are saying.

Output of the curl command returns:

$ curl -k -v https://elasticsearch-es-http.default.svc:9200
* About to connect() to elasticsearch-es-http.default.svc port 9200 (#0)      
*   Trying 10.0.133.217...
* Connected to elasticsearch-es-http.default.svc (10.0.133.217) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=*.[Redacted],OU=PositiveSSL Wildcard,OU=Domain Control Validated
*       start date: Nov 14 00:00:00 2018 GMT
*       expire date: Nov 13 23:59:59 2020 GMT
*       common name: *.[Redacted]
*       issuer: CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: elasticsearch-es-http.default.svc:9200
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< content-type: application/json; charset=UTF-8
< content-length: 459
<
* Connection #0 to host elasticsearch-es-http.default.svc left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

There is an error in the ECK logs:

{
  "log.level": "error",
  "@timestamp": "2020-06-02T14:14:28.568Z",
  "log.logger": "controller-runtime.controller",
  "message": "Reconciler error",
  "service.version": "1.1.1-f13b6d26",
  "service.type": "eck",
  "ecs.version": "1.4.0",
  "controller": "elasticsearch-controller",
  "request": "default/elasticsearch",
  "error": "failed to revert to basic: Post https://elasticsearch-es-http.default.svc:9200/_license/start_basic?acknowledge=true: x509: certificate signed by unknown authority",
  "errorCauses": [
    {
      "error": "failed to revert to basic: Post https://elasticsearch-es-http.default.svc:9200/_license/start_basic?acknowledge=true: x509: certificate signed by unknown authority",
      "errorVerbose": "Post https://elasticsearch-es-http.default.svc:9200/_license/start_basic?acknowledge=true: x509: certificate signed by unknown authority\nfailed to revert to basic\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/license.startBasic\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/license/apply.go:78\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/license.applyLinkedLicense\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/license/apply.go:52\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/license.Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/license/reconcile.go:24\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver.(*defaultDriver).Reconcile.func1\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver/driver.go:203\ngithub.com/elastic/cloud-on-k8s/pkg/controller/common/reconciler.(*Results).Apply\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/common/reconciler/results.go:103\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver.(*defaultDriver).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver/driver.go:196\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).internalReconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:308\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
    }
  ],
  "error.stack_trace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88"
}

This makes me think that the known issue that was fixed in 1.1.2 doesn't treat ComodoCA as a known authority?

Was there a workaround pre-1.1.2 that people are using for these kinds of issues?

Did you set the ca.crt key in the elasticsearch-es-cert Secret ? If the cert has been issued by a known certificate authority it should be empty (or it should not exist).

1 Like

Created a new secret excluding the ca.crt

$ kubectl create secret generic revised-elasticsearch-es-cert --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key
secret/revised-elasticsearch-es-cert created

Added the new revised-elasticsearch-es-cert to my elasticsearch and kibana yaml files and re-apply'd in Kubernetes.

It configured my pods correctly with the new cert and re-created the kibana pod. The Kibana instance started right up, I can connect to it, it is SSL secured.

Thank you and the team so much for the help! Really appreciate it!

1 Like