SSL handshake fails between Kibana/APMServer and Elasticsearch after custom certificate setting on Elasticsearch

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I'm trying to configure our own certificate to the Elasticsearch, Kibana and APM Server, and got ssl handshake errors.

I referenced This document for setting.

The symptoms and my environments are as follow, and if there is any more information please let me know.

Environment

  • ECK Version : 1.0.0Beta1
  • Elastic Image Version : 7.4.2
  • Certificate Info : Issued by a well-known CA to the *.mycompany.com domain
  • Certificate is add to the k8s cluster with this name : my-cert
  • Domain names for service are registered to the DNS Server as follow

Symptoms
I can connect with Chrome browser without any certificate error with this url
https://es.mycompany.com:9200

However, Kibana readness fails and the logs are:

{"type":"log","@timestamp":"2019-11-19T07:59:41Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2019-11-19T07:59:43Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://es-cluster-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2019-11-19T07:59:43Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2019-11-19T07:59:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://es-cluster-es-http.default.svc:9200/"}

APMServer passed the readness check but doen't work properly, and the logs are:

2019-11-19T09:27:47.607Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://es-cluster-es-http.default.svc:9200)): Get https://es-cluster-es-http.default.svc:9200: x509: certificate is valid for *.mycompany.com, mycompany.com, not es-cluster-es-http.default.svc
2019-11-19T09:27:47.607Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(https://es-cluster-es-http.default.svc:9200)) with 149 reconnect attempt(s)




Yaml Files
The followings are the yaml files that I used. (I replaced domain and IP address)

# Source: apm.yaml
apiVersion: apm.k8s.elastic.co/v1beta1
kind: ApmServer
metadata:
  name: es-cluster
spec:
  version: 7.4.2
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 10.0.0.4
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 10.0.0.4
          - dns: apm.mycompany.com
      certificate:
        secretName: my-cert
  count: 1
  elasticsearchRef:
    name: es-cluster
  podTemplate:
    metadata:
      labels:
        project: paas
        idc: 
        app : apmServer
    spec:
      containers:
        - name: apm-server
          resources:
            request:
              memory: 12Gi
              cpu: 1
            limits:
              memory: 12Gi
              cpu: 3
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms6g -Xmx6g"
---
# Source:  es.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: es-cluster
spec:
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 10.0.0.2
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 10.0.0.2
          - dns: es.mycompany.com
      certificate:
        secretName: my-cert
  version: 7.4.2
  nodeSets:
    - name: node
      count: 1
      config:
        node.master: true
        node.ingest: true
        node.data: true
        node.store.allow_mmap: true
      podTemplate:
        metadata:
          labels:
            name: master
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                 privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
            - name: install-gcs-plugins
              command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-gcs']
            - name: install-hdfs-plugins
              command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-hdfs']
          containers:
            - name: elasticsearch
              resources:
                request:
                  memory: 12Gi
                  cpu: 1
                limits:
                  memory: 12Gi
                  cpu: 3
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms6g -Xmx6g"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 500Gi
            storageClassName: fast
---
# Source: kb.yaml
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: es-cluster
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: es-cluster
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 10.0.0.3
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 10.0.0.3
          - dns: kb.mycompany.com
      certificate:
        secretName: my-cert
  podTemplate:
    metadata:
      labels:
        name: kb-alpha
    spec:
      containers:
        - name: kibana
          resources:
            request:
              memory: 12Gi
              cpu: 1
            limits:
              memory: 12Gi
              cpu: 3
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms6g -Xmx6g"

I tried these Elasticsearch http configs and it all fail with same error 
  http:
    ...
    tls:
      selfSignedCertificate:
        disabled: true
      certificate:
        secretName: my-cert


  http:
    ...
    tls:
      certificate:
        secretName: my-cert
2

The above indicates IMO what is going wrong here.

IIUC you have to use a certificate that has $CLUSTER_NAME-es-http.$NAMESPACE and $CLUSTER_NAME-es-http.$NAMESPACE.svc as subjectAltNames for this setup to work. These are the k8s-internal DNS names for the HTTP service ECK creates for each Elasticsearch cluster.

The other stack applications like Kibana and APM service will attempt to contact Elasticsearch through said service.

1 Like
3

Hello @pebrc

Thank you very much for your comment.

I have a follow up question.

According to your reply, I need to add the two k8s internal DNS names as SAN(subjectAltNames) to my existing Certificate for *.mycompany.com?
This would cost extra money for issuing new certificate from Thawte Root Certificates which is Root CA, and I hope there is another way.
I though, the solution is Reserving static IP and custom domain, but it isn't.

4

I am also in need of an answer on this scenario @pebrc. My setup is exactly as described by @Bingu_Shim, although the issue came about for me when upgrading from 1.0.1 => 1.1.0. With 1.0.1 I am able to supply my own cert for both elastic and kibana and have them both configured with DNS. When trying with 1.1.0, I get the same errors above.

5

Did you ever find a solution to this?

1 Like
6

Can you share your config @lsnyder? The one in the OP should still work, though note that the http.tls.selfSignedCertificate and .certificate settings should not be used at the same time. The provided certificate should still take priority, but we will likely add validation to prevent this configuration in the future (as setting a custom certificate disables generation of self signed certs).

Secondly, Kibana should work with custom Elasticsearch certificates, but APM will either need the host names to match or for TLS to be disabled completely. Kibana supports validating everything in the certificate but the host name, while APM does not yet. If you can share the specifics of your issue we can try and troubleshoot it anew.

7

No I was not able to find the solution.

8

My setup does not include APM, it is just Elastic + Kibana
Here is my config for elastic and kibana:

# elastic.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-prd-uks
spec:
  auth: {}
  http:
    service:
      metadata:
        creationTimestamp: null
      spec:
        type: LoadBalancer
    tls:
      certificate:
        secretName: <my-secret-name>
  nodeSets:
  - config:
      node.attr.attr_name: attr_value
      node.data: true
      node.ingest: true
      node.master: true
      node.ml: true
      xpack.monitoring.collection.enabled: true
      xpack.security.authc.realms:
        native:
          native1:
            order: 1
    count: 3
    name: default
    podTemplate:
      metadata:
        creationTimestamp: null
        labels:
          foo: bar
      spec:
        containers:
        - env:
          - name: ES_JAVA_OPTS
            value: -Xms6g -Xmx6g
          name: elasticsearch
          resources:
            limits:
              cpu: 7350m
              memory: 11Gi
            requests:
              cpu: 7350m
              memory: 11Gi
        initContainers:
        - command:
          - sh
          - -c
          - sysctl -w vm.max_map_count=262144
          name: sysctl
          resources: {}
          securityContext:
            privileged: true
        - command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-azure
          name: install-plugins
          resources: {}
    volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
      status: {}
  secureSettings:
  - secretName: <redacted>
  - secretName: <redacted>
  transport:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
  updateStrategy:
    changeBudget: {}
  version: 7.6.2


# kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-prd-uks
  namespace: default
spec:
  count: 1
  elasticsearchRef:
    name: elasticsearch-prd-uks
  http:
    service:
      metadata: {}
      spec:
        type: LoadBalancer
    tls:
      certificate:
        secretName: <my-secret-name>
  podTemplate:
    metadata: {}
    spec: {}
  version: 7.6.2

Kibana Logs:

{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nHEAD https://elasticsearch-prd-uks-es-http.default.svc:9200/.apm-agent-configuration => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-prd-uks-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-prd-uks-es-http.default.svc:9200/_xpack => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}

{"type":"log","@timestamp":"2020-05-07T17:16:54Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:16:56Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:16:56Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:16:59Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:16:59Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:17:01Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
9

I just tried this configuration with new released version

  • elastic operator : 1.1.0
  • Elasticsearch, Kibana : 7.6.2

And got same error as you got

Here is kibana error

2020-05-09T08:24:14.077382373Z {"type":"log","@timestamp":"2020-05-09T08:24:14Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"case\" is disabled."}
2020-05-09T08:25:29.443788733Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [37] plugins: [infra,taskManager,siem,licensing,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
2020-05-09T08:25:29.444798641Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","infra"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.447690701Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","taskManager"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.7385755Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","siem"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.739647503Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","licensing"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.744117332Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","encryptedSavedObjects"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.745404276Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":7,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
2020-05-09T08:25:29.84346468Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","code"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.844729461Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","usageCollection"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.847087285Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","metrics"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.847812528Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","canvas"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.853667359Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","timelion"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.854613642Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","features"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.855590132Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","security"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.874897033Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","apm_oss"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.875407265Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","translations"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.876092198Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","data"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.938860566Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","share"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.940414124Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","home"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.945554172Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","spaces"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.950075428Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","cloud"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.95119985Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","apm"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.955940613Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","graph"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.958959265Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","bfetch"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.965921138Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
2020-05-09T08:25:30.061930512Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","data"],"pid":7,"message":"Request error, retrying\nHEAD https://quickstart-es-http.default.svc:9200/.apm-agent-configuration => unable to verify the first certificate"}
2020-05-09T08:25:30.135797692Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","admin"],"pid":7,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
2020-05-09T08:25:30.142034057Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","data"],"pid":7,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_xpack => unable to verify the first certificate"}
2020-05-09T08:25:30.150702631Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.151361924Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.153163439Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","savedobjects-service"],"pid":7,"message":"Unable to retrieve version information from Elasticsearch nodes."}
2020-05-09T08:25:30.158642587Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.159443419Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.160227781Z Could not create APM Agent configuration: No Living connections
2020-05-09T08:25:30.161510651Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.162179092Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.162943621Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
2020-05-09T08:25:32.5542814Z {"type":"log","@timestamp":"2020-05-09T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:32.554928902Z {"type":"log","@timestamp":"2020-05-09T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:35.052432592Z {"type":"log","@timestamp":"2020-05-09T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:35.052826311Z {"type":"log","@timestamp":"2020-05-09T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:37.555127441Z {"type":"log","@timestamp":"2020-05-09T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:37.555801219Z {"type":"log","@timestamp":"2020-05-09T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:40.058631235Z {"type":"log","@timestamp":"2020-05-09T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:40.05908746Z {"type":"log","@timestamp":"2020-05-09T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:42.555943297Z {"type":"log","@timestamp":"2020-05-09T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:42.556386251Z {"type":"log","@timestamp":"2020-05-09T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:45.059285085Z {"type":"log","@timestamp":"2020-05-09T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:45.059762989Z {"type":"log","@timestamp":"2020-05-09T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:47.559567891Z {"type":"log","@timestamp":"2020-05-09T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:47.560105532Z {"type":"log","@timestamp":"2020-05-09T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:50.060813521Z {"type":"log","@timestamp":"2020-05-09T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:50.061180134Z {"type":"log","@timestamp":"2020-05-09T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}

And this is yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: <my ip>
    tls:
      certificate:
        secretName: <my secret>
  version: 7.6.2
  nodeSets:
    - name: node
      count: 2
      config:
        node.master: true
        node.ingest: true
        node.data: true
        node.store.allow_mmap: true
      podTemplate:
        metadata:
          labels:
            name: node
          annotations:
            "co.elastic.logs/module": elasticsearch
            "co.elastic.metrics/module": elasticsearch
            "co.elastic.metrics/period": "10s"
            "co.elastic.metrics/hosts": "${data.host}:80"
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 1
                limits:
                  memory: 4Gi
                  cpu: 1
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms2g -Xmx2g"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 200Gi
            storageClassName: standard
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.6.2
  count: 1
  elasticsearchRef:
    name: quickstart
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: <my ip>
    tls:
      certificate:
        secretName: <my secrets>
  podTemplate:
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 2Gi
              cpu: 500m
            limits:
              memory: 2Gi
              cpu: 500m
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms1g -Xmx1g"

And when I connect to Elasticsearch with Chrome, there is no warning with the certificate.
image

10

Any further guidance on this issue for @Bingu_Shim and I?

11

Are you including the CA in your custom certificate? It is possible that the CA is in the chrome certificate store already but not in the Elasticsearch image's. You can also try setting logging.verbose: true in the Kibana config to see if there are additional logs.

12

Yes, I have included the intermediate certificate.

And here is the verbose level logs, there was no additional information for the error.

I guess the problem is Kibana is connecting to the the Elasticsearch using Kubernetes internal domain which is different with my Custom Certificate.

There would be two solution for this problem, i guess

  1. Adding an additional Ingress that handle external domain and certificate
  2. Change Kibana to connect Elasticsearch using external domain, as described HERE 
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","plugins","bfetch"],"pid":6,"message":"Initializing plugin"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["info","plugins","bfetch"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: usageCollection"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,cloud"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","legacy-service"],"pid":6,"message":"setting up legacy service"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","root"],"pid":6,"message":"starting root"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","server"],"pid":6,"message":"starting server"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","savedobjects-service"],"pid":6,"message":"Starting SavedObjects service"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: migrations"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nHEAD https://quickstart-es-http.default.svc:9200/.apm-agent-configuration => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_xpack => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
Could not create APM Agent configuration: No Living connections
13

I guess the problem is Kibana is connecting to the the Elasticsearch using Kubernetes internal domain which is different with my Custom Certificate.

We configure Kibana to skip verifying the host name by default (elasticsearch.ssl.verificationMode: certificate) to enable this use case. Since you are receiving an error it cannot validate the chain, may be worth double checking that you have the whole CA chain provided and it validates on its own (without any pre-installed certificates).

14

Hello, Anya_Sabo

You are right, it solved.
I misconfigured the certificate.

Thank you very much~!!

15

@Bingu_Shim what was the misconfiguration on your part? I'm still dealing with the issue. The verbose logging didn't expose any additional errors and I can see from the cert being served in chrome that the intermediate is there too.

16

@lsnyder,

I have to set following following 3 certificates

  • RootCA.crt, ChainCA1.crt, ChainCA2.crt

But, forgot to add ChainCA2.crt when I create Secret.

17

@Anya_Sabo I'm noticing slightly different kibana config options between 1.0 and 1.1, could this be the source of my issue?

1.1

ssl:
    certificateAuthorities: /usr/share/kibana/config/elasticsearch-certs/ca.crt
    verificationMode: certificate

1.0

ssl:
    verificationMode: certificate

1.0 does not contain a path to CA. I'm guessing as you said the issue i'm hitting on 1.1 is that my CA is not included in that certificateAuthorities location. My CA and certs are all bundled into the same pem which was used to create my secret using:

kubectl create secret generic <cert-name> --from-file=tls.crt=<name>.pem --from-file=tls.key=<name>.key

Any thoughts or guidance from here? Why would it work in 1.0 and not in 1.1. The cert i'm using is the exact same cert, created in the exact same method.

18

@lsnyder if you can try adding a ca.crt key to your secret (with the entire CA chain) as described here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html#k8s-setting-up-your-own-certificate
it should populate it correctly. There may have been an inadvertent change between 1.0.1 and 1.1.0 to begin setting that, but I'm not sure off the top of my head and would need to look into it further.

19

Thanks for reporting and pointing us in the right direction @lsnyder. It looks like your issue in particular was a regression in 1.1.0. We tracked it down in https://github.com/elastic/cloud-on-k8s/issues/3082 and a fix should be available in the next release. In the meantime adding the CA explicitly should resolve it. Sorry about that.

20

Awesome! Thank you for staying on top of this and helping us out. I can confirm that explicitly supplying the CA when creating the secret worked. We were able to get everything working last night by doing so. :+1: