Enable TLS between ES and Kibana on an existing cluster

[xcompass (Pan Luo)]

Hi,

I have a running ECK cluster (ES+Kibana) provisioned with http.tls.selfSignedCertificate.disabled: true.
Now, I would like to try out alerts, which requires TLS to be enabled. I tried to change the above settings to:

    http:
      tls:
        selfSignedCertificate:
          #disabled: true
          # add a list of SANs into the self-signed HTTP certificate
          subjectAltNames:
          - dns: elasticsearch-logging-es-http
          - dns: elasticsearch-logging-es-http.default.svc
          - dns: elasticsearch-logging-es-http.default.svc.cluster.local

and did kubectl apply -f es.yaml. The es cluster still on http.

▶ k describe svc elasticsearch-logging-es-http -n logging
Name:              elasticsearch-logging-es-http
Namespace:         logging
Labels:            common.k8s.elastic.co/type=elasticsearch
                   elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-logging
Annotations:       <none>
Selector:          common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-logging
Type:              ClusterIP
IP:                10.3.0.156
Port:              http  9200/TCP
TargetPort:        9200/TCP
Endpoints:         10.2.115.5:9200,10.2.117.5:9200,10.2.39.5:9200
Session Affinity:  None
Events:            <none>

Also tried to delete the svc and it was recreated with http again.

Anyway I can enable TLS on an existing cluster? Thanks.

[sebgl (Sebastien Guilloux)]

Hey @xcompass,
What you did looks correct. The service keeps its name *-es-http though (it is still the HTTP protocol, even though encrypted with TLS), but Elasticsearch itself should only be reachable through HTTPS. You can curl the endpoint using https to verify that's the case.

[xcompass (Pan Luo)]

Thanks @sebgl for the reply. I actually did test https with curl, and got the following error:

[root@elasticsearch-logging-es-default-2 elasticsearch]# curl https://localhost:9200
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

If I use http, I got correct response:

[root@elasticsearch-logging-es-default-2 elasticsearch]# curl http://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

I did more testing and found out I have to change the following to force ES to be on https:

spec:
  nodeSets:
    - name: default
      config:
        xpack.security.http.ssl.enabled: true # I know this is not recommended
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              env:
                - name: READINESS_PROBE_PROTOCOL
                  value: "https"

But the svc still use shows http under port as shown in my original post. (I understand the svc name contains http and it doesn't matter)

Port:              http  9200/TCP

I deployed a new "quickstart" cluster and the svc shown as https:

▶ k describe svc quickstart-es-http
Name:              quickstart-es-http
Namespace:         default
Labels:            common.k8s.elastic.co/type=elasticsearch
                   elasticsearch.k8s.elastic.co/cluster-name=quickstart
Annotations:       <none>
Selector:          common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=quickstart
Type:              ClusterIP
IP:                10.3.0.171
Port:              https  9200/TCP
TargetPort:        9200/TCP
Endpoints:         10.2.10.8:9200
Session Affinity:  None
Events:            <none>

I think this causes kibana to connect ES using http instead of https. Is there a way to force operator generate https on the svc so kibana can get the correct reference? Or is it a bug that operator didn't pick up the change and generate the correct svc?

[xcompass (Pan Luo)]

I was able to fix the issue by setting

    http:
      tls:
        selfSignedCertificate:
          disabled: false

explicitly. No other setting needed. Not sure why it didn't work last night. I must miss something.