SSL handshake fails between Kibana/APMServer and Elasticsearch after custom certificate setting on Elasticsearch

I'm trying to configure our own certificate to the Elasticsearch, Kibana and APM Server, and got ssl handshake errors.

I referenced This document for setting.

The symptoms and my environments are as follow, and if there is any more information please let me know.

Environment

  • ECK Version : 1.0.0Beta1
  • Elastic Image Version : 7.4.2
  • Certificate Info : Issued by a well-known CA to the *.mycompany.com domain
  • Certificate is add to the k8s cluster with this name : my-cert
  • Domain names for service are registered to the DNS Server as follow

Symptoms
I can connect with Chrome browser without any certificate error with this url
https://es.mycompany.com:9200

However, Kibana readness fails and the logs are:

{"type":"log","@timestamp":"2019-11-19T07:59:41Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2019-11-19T07:59:43Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://es-cluster-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2019-11-19T07:59:43Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2019-11-19T07:59:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://es-cluster-es-http.default.svc:9200/"}

APMServer passed the readness check but doen't work properly, and the logs are:

2019-11-19T09:27:47.607Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://es-cluster-es-http.default.svc:9200)): Get https://es-cluster-es-http.default.svc:9200: x509: certificate is valid for *.mycompany.com, mycompany.com, not es-cluster-es-http.default.svc
2019-11-19T09:27:47.607Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(https://es-cluster-es-http.default.svc:9200)) with 149 reconnect attempt(s)




Yaml Files
The followings are the yaml files that I used. (I replaced domain and IP address)

# Source: apm.yaml
apiVersion: apm.k8s.elastic.co/v1beta1
kind: ApmServer
metadata:
  name: es-cluster
spec:
  version: 7.4.2
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 10.0.0.4
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 10.0.0.4
          - dns: apm.mycompany.com
      certificate:
        secretName: my-cert
  count: 1
  elasticsearchRef:
    name: es-cluster
  podTemplate:
    metadata:
      labels:
        project: paas
        idc: 
        app : apmServer
    spec:
      containers:
        - name: apm-server
          resources:
            request:
              memory: 12Gi
              cpu: 1
            limits:
              memory: 12Gi
              cpu: 3
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms6g -Xmx6g"
---
# Source:  es.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: es-cluster
spec:
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 10.0.0.2
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 10.0.0.2
          - dns: es.mycompany.com
      certificate:
        secretName: my-cert
  version: 7.4.2
  nodeSets:
    - name: node
      count: 1
      config:
        node.master: true
        node.ingest: true
        node.data: true
        node.store.allow_mmap: true
      podTemplate:
        metadata:
          labels:
            name: master
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                 privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
            - name: install-gcs-plugins
              command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-gcs']
            - name: install-hdfs-plugins
              command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-hdfs']
          containers:
            - name: elasticsearch
              resources:
                request:
                  memory: 12Gi
                  cpu: 1
                limits:
                  memory: 12Gi
                  cpu: 3
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms6g -Xmx6g"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 500Gi
            storageClassName: fast
---
# Source: kb.yaml
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: es-cluster
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: es-cluster
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 10.0.0.3
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 10.0.0.3
          - dns: kb.mycompany.com
      certificate:
        secretName: my-cert
  podTemplate:
    metadata:
      labels:
        name: kb-alpha
    spec:
      containers:
        - name: kibana
          resources:
            request:
              memory: 12Gi
              cpu: 1
            limits:
              memory: 12Gi
              cpu: 3
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms6g -Xmx6g"


I tried these Elasticsearch http configs and it all fail with same error
  http:
    ...
    tls:
      selfSignedCertificate:
        disabled: true
      certificate:
        secretName: my-cert

  http:
    ...
    tls:
      certificate:
        secretName: my-cert

The above indicates IMO what is going wrong here.

IIUC you have to use a certificate that has $CLUSTER_NAME-es-http.$NAMESPACE and $CLUSTER_NAME-es-http.$NAMESPACE.svc as subjectAltNames for this setup to work. These are the k8s-internal DNS names for the HTTP service ECK creates for each Elasticsearch cluster.

The other stack applications like Kibana and APM service will attempt to contact Elasticsearch through said service.

1 Like

Hello @pebrc

Thank you very much for your comment.

I have a follow up question.

According to your reply, I need to add the two k8s internal DNS names as SAN(subjectAltNames) to my existing Certificate for *.mycompany.com?
This would cost extra money for issuing new certificate from Thawte Root Certificates which is Root CA, and I hope there is another way.
I though, the solution is Reserving static IP and custom domain, but it isn't.