Received plaintext http traffic on an https channel, closing connection

I have deployed ECK (using helm) on my k8s cluster and i am attempting to install elasticsearch following the docs. Deploy an Elasticsearch cluster | Elastic Cloud on Kubernetes [1.5] | Elastic

I have externally exposed service/elasticsearch-prod-es-http so that i can connect to it from outside of my k8s cluster. However as you can see when i try to connect to it either from curl or the browser i receive an error "502 Bad Gateway" error.

curl elasticsearch.dev.acme.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>

Upon checking the pod (elasticsearch-prod-es-default-0) i can see the following message repeated.

{"type": "server", "timestamp": "2021-04-27T13:12:20,048Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch-prod", "node.name": "elasticsearch-prod-es-default-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.0.5.81:9200, remoteAddress=/10.0.3.50:46380}", "cluster.uuid": "t0mRfv7kREGQhXW9DVM3Vw", "node.id": "nCyAItDmSqGZRa3lApsC6g" }

Can you help me understand why this is occuring and how to fix it?

I suspect it has something to do with my TLS configuration because when i disable TLS, im able to connect to it externally without issues. However in a production environment i think keeping TLS enabled is important?

FYI i am able to port-forward the service and connect to it with curl using the -k flag.

What i have tried

  1. I have tried adding my domain to the section as described here HTTP settings and TLS SANs | Elastic Cloud on Kubernetes [1.5] | Elastic
  2. I have tried using openssl to generate a self signed certificate but that did not work. Trying to connect locally returns the following error message.

curl -u "elastic:$PASSWORD" "https://localhost:9200"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

  1. I have tried generating a certificate using the tool Encrypting communications in Elasticsearch | Elasticsearch Guide [7.9] | Elastic

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.12 --pem

Then using the .crt and .key generated i created a kubectl secret elastic-tls-cert. But again curling localhost without -k gave the following error:

curl --cacert cacert.pem -u "elastic:$PASSWORD" -XGET "https://localhost:9200"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

elasticsearch.yml

# This sample sets up an Elasticsearch cluster with 3 nodes.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-prod
  namespace: elastic-system
spec:
  version: 7.12.0
  nodeSets:
  - name: default
    config:
      # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
      node.roles: ["master", "data", "ingest", "ml"]
      # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
      node.store.allow_mmap: false
      xpack.security.enabled: true
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          foo: bar
      spec:
        nodeSelector: 
          acme/node-type: ops

        # this changes the kernel setting on the node to allow ES to use mmap
        # if you uncomment this init container you will likely also want to remove the
        # "node.store.allow_mmap: false" setting above
        # initContainers:
        # - name: sysctl
        #   securityContext:
        #     privileged: true
        #   command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        ###
        # uncomment the line below if you are using a service mesh such as linkerd2 that uses service account tokens for pod identification.
        # automountServiceAccountToken: true
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 4Gi
              cpu: 1
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
    count: 3
  #   # request 2Gi of persistent data storage for pods in this topology element
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 250Gi
        storageClassName: elasticsearch
  # # inject secure settings into Elasticsearch nodes from k8s secrets references
  # secureSettings:
  # - secretName: ref-to-secret
  # - secretName: another-ref-to-secret
  #   # expose only a subset of the secret keys (optional)
  #   entries:
  #   - key: value1
  #     path: newkey # project a key to a specific path (optional)
  http:
    service:
      spec:
        # expose this cluster Service with a LoadBalancer
        type: NodePort
    # tls:
      # selfSignedCertificate:
        # add a list of SANs into the self-signed HTTP certificate
        subjectAltNames:
        # - ip: 192.168.1.2
        # - ip: 192.168.1.3
        # - dns: elasticsearch.dev.acme.com
        # - dns: localhost
      # certificate:
      #   # provide your own certificate
      #   secretName: elastic-tls-cert

kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

helm list

    NAME            	NAMESPACE     	REVISION	UPDATED                                	STATUS  	CHART             	APP VERSION
elastic-operator	elastic-system	1       	2021-04-26 11:18:02.286692269 +0100 BST	deployed	eck-operator-1.5.0	1.5.0      

resources

pod/elastic-operator-0                1/1     Running   0          4h58m   10.0.5.142   ip-10-0-5-71.us-east-2.compute.internal    <none>           <none>
pod/elasticsearch-prod-es-default-0   1/1     Running   0          9m5s    10.0.5.81    ip-10-0-5-71.us-east-2.compute.internal    <none>           <none>
pod/elasticsearch-prod-es-default-1   1/1     Running   0          9m5s    10.0.1.128   ip-10-0-1-207.us-east-2.compute.internal   <none>           <none>
pod/elasticsearch-prod-es-default-2   1/1     Running   0          9m5s    10.0.5.60    ip-10-0-5-71.us-east-2.compute.internal    <none>           <none>

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE    SELECTOR
service/elastic-operator-webhook          ClusterIP   172.20.218.208   <none>        443/TCP          26h    app.kubernetes.io/instance=elastic-operator,app.kubernetes.io/name=elastic-operator
service/elasticsearch-prod-es-default     ClusterIP   None             <none>        9200/TCP         9m5s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod,elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-prod-es-default
service/elasticsearch-prod-es-http        NodePort    172.20.229.173   <none>        9200:30604/TCP   9m6s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod
service/elasticsearch-prod-es-transport   ClusterIP   None             <none>        9300/TCP         9m6s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod

aws alb ingress controller

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: elastic-ingress
  namespace: elastic-system
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: "<redacted>"
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: <redacted>
    alb.ingress.kubernetes.io/tags: Environment=prod,Team=dev
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '300'
    alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=acme-aws-ingress-logs,access_logs.s3.prefix=dev-ingress
spec:
  rules:
    - host: elasticsearch.dev.acme.com
      http:
        paths:
          - path: /*
            pathType: Prefix
            backend:
              service:
                name: elasticsearch-prod-es-http
                port:
                  number: 9200
    # - host: kibana.dev.acme.com
    #   http:
    #     paths:
    #       - path: /*
    #         pathType: Prefix
    #         backend:
    #           service:
    #             name: kibana-prod-kb-http
    #             port:
    #               number: 5601

I'm posting the solution as we solved this issue on the Elastic Stack Community Slack.

I suspect it has something to do with my TLS configuration because when i disable TLS, im able to connect to it externally without issues. However in a production environment i think keeping TLS enabled is important?

Yes, it's definitively better to keep TLS.

I have tried using openssl to generate a self signed certificate but that did not work. Trying to connect locally returns the following error message.

There is a confusion here. It is not mandatory to configure a TLS certificate. By default, ECK manages all certificates for you. But if you want, you can bring your own certificate.

When using an Ingress, the error received plaintext http traffic on an https channel is often caused by the ingress not being configured correctly to force HTTPS and not allow HTTP.

The configuration then depends on the type of ingress you are using. For example:

  • AWS ALB Ingress uses: alb.ingress.kubernetes.io/backend-protocol: HTTPS.
  • GKE Ingress uses: kubernetes.io/ingress.allow-http: "false".

A full example to illustrate how to expose an Elasticsearch and Kibana instance to the outside world using custom TLS certificates and using a Google Cloud Load Balancer (GCLB) is available here.