# ECK managed elasticsearch (with TLS enabled) getting flooded with "received plaintext http traffic on an https channel" logs

**URL:** <https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117>\
**Category:** Elasticsearch\
**Created:** [December 13, 2022, 11:30am UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117 "2022-12-13T11:30:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Daithi\_O\_Conchobhair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daithi_o_conchobhair/32/112206_2.png) [@Daithi\_O\_Conchobhair](https://discuss.elastic.co/u/Daithi_O_Conchobhair)\
**Post date:** [December 13, 2022, 11:30am UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117/1 "2022-12-13T11:30:56Z")

</div>

Hi there,

This is such a strange issue I am not sure if it is a defect or something that I am doing wrong. I will say in my defence that I see it when I just use the [quickstart guide](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html#k8s-deploy-elasticsearch) also.

Versions:

1. IBM Kubernetes Service version 1.25.4
2. ECK Operator version 2.5.0
3. Elasticsearch version 8.5.3

Steps to reproduce:

1. [Install the operator](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-install-helm.html#k8s-install-helm-global) with the helm charts using all default values. `helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace`
2. [Install the quickstart guide](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html#k8s-deploy-elasticsearch) elasticsearch cluster in the `elastic-stack` namespace. I have modified the quickstart guide to workaround an [open issue](https://github.com/elastic/cloud-on-k8s/issues/6156) which an ES engineer is helping me investigate.

```auto
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.5.3
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          runAsGroup: 0
        initContainers:
        - name: elastic-internal-init-filesystem
          securityContext:
            runAsUser: 0
            runAsGroup: 0
EOF

```

1. Wait for the `quickstart-es-default-0` to be ready and running for about 5 minutes.
2. Check the logs of `quickstart-es-default-0`. `kubectl logs -f quickstart-es-default-0`:

```auto
{"@timestamp":"2022-12-13T11:30:02.444Z", "log.level": "INFO", "message":"successfully downloaded geoip database [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][generic][T#13]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:02.993Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][generic][T#20]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:30.070Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50708}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#6]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:30.081Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50718}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#7]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:31.072Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50918}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#9]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:31.076Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50922}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#10]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:32.044Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51074}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#11]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:32.062Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51080}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#12]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:33.064Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51248}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#13]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:33.069Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51264}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#14]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:34.069Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51418}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#15]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:34.080Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51422}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#16]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:35.050Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51568}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#17]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:35.060Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51572}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#18]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}

```

Expected behaviour:

1. I don't expect to see the http messages. There is no Kibana or any other services connecting to ES at this point, the service is not exposed outside the cluster.

I am not sure how to debug this issue any further. It looks like the erroneous connections are originating from within the ES pod itself? Remote Address is always 127.0.0.1 but with a different port.

Any help would be greatly appreciated - Is it possible to set the log level for `org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport` to `ERROR` while I debug this? It is killing my logs.

---

<div class="post-metadata">

**Author:** ![Daithi\_O\_Conchobhair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daithi_o_conchobhair/32/112206_2.png) [@Daithi\_O\_Conchobhair](https://discuss.elastic.co/u/Daithi_O_Conchobhair)\
**Post date:** [December 13, 2022, 11:42am UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117/2 "2022-12-13T11:42:47Z")

</div>

P.S. Logs available as Gists:

[quickstart-es-default-0.log](https://gist.github.com/dobharweim/e36bae540cc736f05198ab95ad697ed5)  
[elastic-operator-0.log](https://gist.github.com/dobharweim/a29be26407c3dcbde9dda108012e5051)

I see something peculiar in the quickstart-es-default-0.log:  
`Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file `

Is this expected? The file looks ok but I do two license types under `xpack.license.upload.types`:

```auto
cluster:
  name: quickstart
  routing:
    allocation:
      awareness:
        attributes: k8s_node_name
discovery:
  seed_hosts: []
  seed_providers: file
http:
  publish_host: ${POD_NAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc
network:
  host: "0"
  publish_host: ${POD_IP}
node:
  attr:
    k8s_node_name: ${NODE_NAME}
  name: ${POD_NAME}
  store:
    allow_mmap: false
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
xpack:
  license:
    upload:
      types:
      - trial
      - enterprise
  security:
    authc:
      realms:
        file:
          file1:
            order: -100
        native:
          native1:
            order: -99
      reserved_realm:
        enabled: "false"
    enabled: "true"
    http:
      ssl:
        certificate: /usr/share/elasticsearch/config/http-certs/tls.crt
        certificate_authorities: /usr/share/elasticsearch/config/http-certs/ca.crt
        enabled: true
        key: /usr/share/elasticsearch/config/http-certs/tls.key
    transport:
      ssl:
        certificate: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.crt
        certificate_authorities:
        - /usr/share/elasticsearch/config/transport-certs/ca.crt
        - /usr/share/elasticsearch/config/transport-remote-certs/ca.crt
        enabled: "true"
        key: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.key
        verification_mode: certificate

```

---

<div class="post-metadata">

**Author:** ![Daithi\_O\_Conchobhair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daithi_o_conchobhair/32/112206_2.png) [@Daithi\_O\_Conchobhair](https://discuss.elastic.co/u/Daithi_O_Conchobhair)\
**Post date:** [December 13, 2022, 11:52am UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117/3 "2022-12-13T11:52:05Z")

</div>

My `/usr/share/elasticsearch/config/elasticsearch.yml` is identical on my GKE deployment but I'm not getting the same `Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file` message in my GKE pod!

---

<div class="post-metadata">

**Author:** ![Daithi\_O\_Conchobhair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daithi_o_conchobhair/32/112206_2.png) [@Daithi\_O\_Conchobhair](https://discuss.elastic.co/u/Daithi_O_Conchobhair)\
**Post date:** [December 13, 2022, 12:09pm UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117/4 "2022-12-13T12:09:17Z")

</div>

IKS env vars used in elasticsearch.yml:

```auto
POD_NAME: quickstart-es-default-0
HEADLESS_SERVICE_NAME: quickstart-es-default
NAMESPACE: elastic-stack
POD_IP: 172.30.147.97
NODE_NAME: 10.135.121.63

```

GKE env vars used in elasticsearch.yml:

```auto
POD_NAME: quickstart-es-default-0
HEADLESS_SERVICE_NAME: quickstart-es-default
NAMESPACE: default
POD_IP: 10.8.0.133
NODE_NAME: gk3-autopilot-cluster-1-default-pool-d3333212-cgr2

```

I don't think the env vars difference is causing that strange warning message but could the POD\_IP difference (10. vs a 172. network) be causing the http one?

---

<div class="post-metadata">

**Author:** ![Daithi\_O\_Conchobhair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daithi_o_conchobhair/32/112206_2.png) [@Daithi\_O\_Conchobhair](https://discuss.elastic.co/u/Daithi_O_Conchobhair)\
**Post date:** [December 13, 2022, 12:36pm UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117/5 "2022-12-13T12:36:33Z")

</div>

So I realise that IKS is not on the supported versions list. I spun up an Openshift cluster also as it is on the supported versions list. I did it on IBM cloud. I see similar env vars and yaml file, and I see similar error messages in the logs about the http over a https connection.

```auto
POD_NAME: quickstart-es-default-0
HEADLESS_SERVICE_NAME: quickstart-es-default
NAMESPACE: elastic-system
POD_IP: 172.17.23.80
NODE_NAME: 10.242.128.7

```

`elasticsearch.yaml` is similarly identical to my one on GKE and IKS:

```auto
cluster:
  name: quickstart
  routing:
    allocation:
      awareness:
        attributes: k8s_node_name
discovery:
  seed_hosts: []
  seed_providers: file
http:
  publish_host: ${POD_NAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc
network:
  host: "0"
  publish_host: ${POD_IP}
node:
  attr:
    k8s_node_name: ${NODE_NAME}
  name: ${POD_NAME}
  store:
    allow_mmap: false
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
xpack:
  license:
    upload:
      types:
      - trial
      - enterprise
  security:
    authc:
      realms:
        file:
          file1:
            order: -100
        native:
          native1:
            order: -99
      reserved_realm:
        enabled: "false"
    enabled: "true"
    http:
      ssl:
        certificate: /usr/share/elasticsearch/config/http-certs/tls.crt
        certificate_authorities: /usr/share/elasticsearch/config/http-certs/ca.crt
        enabled: true
        key: /usr/share/elasticsearch/config/http-certs/tls.key
    transport:
      ssl:
        certificate: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.crt
        certificate_authorities:
        - /usr/share/elasticsearch/config/transport-certs/ca.crt
        - /usr/share/elasticsearch/config/transport-remote-certs/ca.crt
        enabled: "true"
        key: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.key
        verification_mode: certificate

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2023, 12:37pm UTC](https://discuss.elastic.co/t/eck-managed-elasticsearch-with-tls-enabled-getting-flooded-with-received-plaintext-http-traffic-on-an-https-channel-logs/321117/6 "2023-01-10T12:37:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.