# Received plaintext http traffic on an https channel, closing connection

**URL:** <https://discuss.elastic.co/t/received-plaintext-http-traffic-on-an-https-channel-closing-connection/271380>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [April 27, 2021, 1:33pm UTC](https://discuss.elastic.co/t/received-plaintext-http-traffic-on-an-https-channel-closing-connection/271380 "2021-04-27T13:33:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [April 27, 2021, 1:33pm UTC](https://discuss.elastic.co/t/received-plaintext-http-traffic-on-an-https-channel-closing-connection/271380/1 "2021-04-27T13:33:40Z")

</div>

I have deployed ECK (using helm) on my k8s cluster and i am attempting to install elasticsearch following the docs. [Deploy an Elasticsearch cluster | Elastic Cloud on Kubernetes [2.10] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html)

I have externally exposed service/elasticsearch-prod-es-http so that i can connect to it from outside of my k8s cluster. However as you can see when i try to connect to it either from curl or the browser i receive an error "502 Bad Gateway" error.

```
curl elasticsearch.dev.acme.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>

```

Upon checking the pod (elasticsearch-prod-es-default-0) i can see the following message repeated.

> {"type": "server", "timestamp": "2021-04-27T13:12:20,048Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch-prod", "node.name": "elasticsearch-prod-es-default-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.0.5.81:9200, remoteAddress=/10.0.3.50:46380}", "cluster.uuid": "t0mRfv7kREGQhXW9DVM3Vw", "node.id": "nCyAItDmSqGZRa3lApsC6g" }

**Can you help me understand why this is occuring and how to fix it?**

I suspect it has something to do with my TLS configuration because when i disable TLS, im able to connect to it externally without issues. However in a production environment i think keeping TLS enabled is important?

FYI i am able to port-forward the service and connect to it with curl using the -k flag.

**What i have tried**

1. I have tried adding my domain to the section as described here [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-http-settings-tls-sans.html#k8s-elasticsearch-http-service-san](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-http-settings-tls-sans.html#k8s-elasticsearch-http-service-san)
2. I have tried using openssl to generate a self signed certificate but that did not work. Trying to connect locally returns the following error message.

> curl -u "elastic:$PASSWORD" "[https://localhost:9200](https://localhost:9200)"  
> curl: (60) SSL certificate problem: unable to get local issuer certificate  
> More details here: [curl - SSL CA Certificates](https://curl.haxx.se/docs/sslcerts.html)  
> curl failed to verify the legitimacy of the server and therefore could not  
> establish a secure connection to it. To learn more about this situation and  
> how to fix it, please visit the web page mentioned above.

1. I have tried generating a certificate using the tool [Encrypting communications in Elasticsearch | Elasticsearch Guide [7.9] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/configuring-tls.html#tls-transport)

> bin/elasticsearch-certutil ca  
> bin/elasticsearch-certutil cert --ca elastic-stack-ca.12 --pem

Then using the .crt and .key generated i created a kubectl secret `elastic-tls-cert`. But again curling localhost without -k gave the following error:

> curl --cacert cacert.pem -u "elastic:$PASSWORD" -XGET "[https://localhost:9200](https://localhost:9200)"  
> curl: (60) SSL certificate problem: unable to get local issuer certificate  
> More details here: [curl - SSL CA Certificates](https://curl.haxx.se/docs/sslcerts.html)  
> curl failed to verify the legitimacy of the server and therefore could not  
> establish a secure connection to it. To learn more about this situation and  
> how to fix it, please visit the web page mentioned above.

**elasticsearch.yml**

```
# This sample sets up an Elasticsearch cluster with 3 nodes.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-prod
  namespace: elastic-system
spec:
  version: 7.12.0
  nodeSets:
  - name: default
    config:
      # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
      node.roles: ["master", "data", "ingest", "ml"]
      # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
      node.store.allow_mmap: false
      xpack.security.enabled: true
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          foo: bar
      spec:
        nodeSelector: 
          acme/node-type: ops

        # this changes the kernel setting on the node to allow ES to use mmap
        # if you uncomment this init container you will likely also want to remove the
        # "node.store.allow_mmap: false" setting above
        # initContainers:
        # - name: sysctl
        # securityContext:
        # privileged: true
        # command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        ###
        # uncomment the line below if you are using a service mesh such as linkerd2 that uses service account tokens for pod identification.
        # automountServiceAccountToken: true
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 4Gi
              cpu: 1
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
    count: 3
  # # request 2Gi of persistent data storage for pods in this topology element
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 250Gi
        storageClassName: elasticsearch
  # # inject secure settings into Elasticsearch nodes from k8s secrets references
  # secureSettings:
  # - secretName: ref-to-secret
  # - secretName: another-ref-to-secret
  # # expose only a subset of the secret keys (optional)
  # entries:
  # - key: value1
  # path: newkey # project a key to a specific path (optional)
  http:
    service:
      spec:
        # expose this cluster Service with a LoadBalancer
        type: NodePort
    # tls:
      # selfSignedCertificate:
        # add a list of SANs into the self-signed HTTP certificate
        subjectAltNames:
        # - ip: 192.168.1.2
        # - ip: 192.168.1.3
        # - dns: elasticsearch.dev.acme.com
        # - dns: localhost
      # certificate:
      # # provide your own certificate
      # secretName: elastic-tls-cert

```

**kubectl version**

```
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

```

**helm list**

```auto
    NAME NAMESPACE REVISION	UPDATED STATUS CHART APP VERSION
elastic-operator	elastic-system	1 2021-04-26 11:18:02.286692269 +0100 BST	deployed	eck-operator-1.5.0	1.5.0      

```

**resources**

```auto
pod/elastic-operator-0 1/1 Running 0 4h58m 10.0.5.142 ip-10-0-5-71.us-east-2.compute.internal <none> <none>
pod/elasticsearch-prod-es-default-0 1/1 Running 0 9m5s 10.0.5.81 ip-10-0-5-71.us-east-2.compute.internal <none> <none>
pod/elasticsearch-prod-es-default-1 1/1 Running 0 9m5s 10.0.1.128 ip-10-0-1-207.us-east-2.compute.internal <none> <none>
pod/elasticsearch-prod-es-default-2 1/1 Running 0 9m5s 10.0.5.60 ip-10-0-5-71.us-east-2.compute.internal <none> <none>

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/elastic-operator-webhook ClusterIP 172.20.218.208 <none> 443/TCP 26h app.kubernetes.io/instance=elastic-operator,app.kubernetes.io/name=elastic-operator
service/elasticsearch-prod-es-default ClusterIP None <none> 9200/TCP 9m5s common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod,elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-prod-es-default
service/elasticsearch-prod-es-http NodePort 172.20.229.173 <none> 9200:30604/TCP 9m6s common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod
service/elasticsearch-prod-es-transport ClusterIP None <none> 9300/TCP 9m6s common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod

```

**aws alb ingress controller**

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: elastic-ingress
  namespace: elastic-system
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: "<redacted>"
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: <redacted>
    alb.ingress.kubernetes.io/tags: Environment=prod,Team=dev
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '300'
    alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=acme-aws-ingress-logs,access_logs.s3.prefix=dev-ingress
spec:
  rules:
    - host: elasticsearch.dev.acme.com
      http:
        paths:
          - path: /*
            pathType: Prefix
            backend:
              service:
                name: elasticsearch-prod-es-http
                port:
                  number: 9200
    # - host: kibana.dev.acme.com
    # http:
    # paths:
    # - path: /*
    # pathType: Prefix
    # backend:
    # service:
    # name: kibana-prod-kb-http
    # port:
    # number: 5601

```

---

<div class="post-metadata">

**Author:** ![Thibault\_Richard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibault_richard/32/50513_2.png) [@Thibault\_Richard](https://discuss.elastic.co/u/Thibault_Richard)\
**Post date:** [April 28, 2021, 12:32pm UTC](https://discuss.elastic.co/t/received-plaintext-http-traffic-on-an-https-channel-closing-connection/271380/2 "2021-04-28T12:32:58Z")

</div>

I'm posting the solution as we solved this issue on the [Elastic Stack Community Slack](https://elasticstack.slack.com/archives/CS9KR083S/p1619453940073700).

> I suspect it has something to do with my TLS configuration because when i disable TLS, im able to connect to it externally without issues. However in a production environment i think keeping TLS enabled is important?

Yes, it's definitively better to keep TLS.

> I have tried using openssl to generate a self signed certificate but that did not work. Trying to connect locally returns the following error message.

There is a confusion here. It is not mandatory to configure a TLS certificate. By default, ECK manages all certificates for you. But if you want, you can [bring your own certificate](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-tls-certificates.html#k8s-setting-up-your-own-certificate).

When using an Ingress, the error `received plaintext http traffic on an https channel` is often caused by the ingress not being configured correctly to force HTTPS and not allow HTTP.

The configuration then depends on the type of ingress you are using. For example:

- AWS ALB Ingress uses: `alb.ingress.kubernetes.io/backend-protocol: HTTPS`.
- GKE Ingress uses: `kubernetes.io/ingress.allow-http: "false"`.

A full example to illustrate how to expose an Elasticsearch and Kibana instance to the outside world using custom TLS certificates and using a Google Cloud Load Balancer (GCLB) is available [here](https://github.com/elastic/cloud-on-k8s/tree/1.5/config/recipes/gclb).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 26, 2021, 12:33pm UTC](https://discuss.elastic.co/t/received-plaintext-http-traffic-on-an-https-channel-closing-connection/271380/3 "2021-05-26T12:33:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.