ECK managed elasticsearch (with TLS enabled) getting flooded with "received plaintext http traffic on an https channel" logs

Elastic Stack Elasticsearch
1

Hi there,

This is such a strange issue I am not sure if it is a defect or something that I am doing wrong. I will say in my defence that I see it when I just use the quickstart guide also.

Versions:

  1. IBM Kubernetes Service version 1.25.4
  2. ECK Operator version 2.5.0
  3. Elasticsearch version 8.5.3

Steps to reproduce:

  1. Install the operator with the helm charts using all default values. helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace
  2. Install the quickstart guide elasticsearch cluster in the elastic-stack namespace. I have modified the quickstart guide to workaround an open issue which an ES engineer is helping me investigate.
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.5.3
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          runAsGroup: 0
        initContainers:
        - name: elastic-internal-init-filesystem
          securityContext:
            runAsUser: 0
            runAsGroup: 0
EOF
  1. Wait for the quickstart-es-default-0 to be ready and running for about 5 minutes.
  2. Check the logs of quickstart-es-default-0. kubectl logs -f quickstart-es-default-0:
{"@timestamp":"2022-12-13T11:30:02.444Z", "log.level": "INFO", "message":"successfully downloaded geoip database [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][generic][T#13]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:02.993Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][generic][T#20]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:30.070Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50708}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#6]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:30.081Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50718}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#7]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:31.072Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50918}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#9]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:31.076Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:50922}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#10]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:32.044Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51074}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#11]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:32.062Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51080}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#12]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:33.064Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51248}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#13]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:33.069Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51264}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#14]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:34.069Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51418}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#15]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:34.080Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51422}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#16]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:35.050Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51568}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#17]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"@timestamp":"2022-12-13T11:30:35.060Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:51572}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#18]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"QoPjTRQ4QZyHTKUrSkXcLA","elasticsearch.node.id":"9R1-dv3WRB-8fDYgqGFUOg","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}

Expected behaviour:

  1. I don't expect to see the http messages. There is no Kibana or any other services connecting to ES at this point, the service is not exposed outside the cluster.

I am not sure how to debug this issue any further. It looks like the erroneous connections are originating from within the ES pod itself? Remote Address is always 127.0.0.1 but with a different port.

Any help would be greatly appreciated - Is it possible to set the log level for org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport to ERROR while I debug this? It is killing my logs.

2

P.S. Logs available as Gists:

quickstart-es-default-0.log
elastic-operator-0.log

I see something peculiar in the quickstart-es-default-0.log:
Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file

Is this expected? The file looks ok but I do two license types under xpack.license.upload.types:

cluster:
  name: quickstart
  routing:
    allocation:
      awareness:
        attributes: k8s_node_name
discovery:
  seed_hosts: []
  seed_providers: file
http:
  publish_host: ${POD_NAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc
network:
  host: "0"
  publish_host: ${POD_IP}
node:
  attr:
    k8s_node_name: ${NODE_NAME}
  name: ${POD_NAME}
  store:
    allow_mmap: false
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
xpack:
  license:
    upload:
      types:
      - trial
      - enterprise
  security:
    authc:
      realms:
        file:
          file1:
            order: -100
        native:
          native1:
            order: -99
      reserved_realm:
        enabled: "false"
    enabled: "true"
    http:
      ssl:
        certificate: /usr/share/elasticsearch/config/http-certs/tls.crt
        certificate_authorities: /usr/share/elasticsearch/config/http-certs/ca.crt
        enabled: true
        key: /usr/share/elasticsearch/config/http-certs/tls.key
    transport:
      ssl:
        certificate: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.crt
        certificate_authorities:
        - /usr/share/elasticsearch/config/transport-certs/ca.crt
        - /usr/share/elasticsearch/config/transport-remote-certs/ca.crt
        enabled: "true"
        key: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.key
        verification_mode: certificate
3

My /usr/share/elasticsearch/config/elasticsearch.yml is identical on my GKE deployment but I'm not getting the same Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file message in my GKE pod!

4

IKS env vars used in elasticsearch.yml:

POD_NAME: quickstart-es-default-0
HEADLESS_SERVICE_NAME: quickstart-es-default
NAMESPACE: elastic-stack
POD_IP: 172.30.147.97
NODE_NAME: 10.135.121.63

GKE env vars used in elasticsearch.yml:

POD_NAME: quickstart-es-default-0
HEADLESS_SERVICE_NAME: quickstart-es-default
NAMESPACE: default
POD_IP: 10.8.0.133
NODE_NAME: gk3-autopilot-cluster-1-default-pool-d3333212-cgr2

I don't think the env vars difference is causing that strange warning message but could the POD_IP difference (10. vs a 172. network) be causing the http one?

5

So I realise that IKS is not on the supported versions list. I spun up an Openshift cluster also as it is on the supported versions list. I did it on IBM cloud. I see similar env vars and yaml file, and I see similar error messages in the logs about the http over a https connection.

POD_NAME: quickstart-es-default-0
HEADLESS_SERVICE_NAME: quickstart-es-default
NAMESPACE: elastic-system
POD_IP: 172.17.23.80
NODE_NAME: 10.242.128.7

elasticsearch.yaml is similarly identical to my one on GKE and IKS:

cluster:
  name: quickstart
  routing:
    allocation:
      awareness:
        attributes: k8s_node_name
discovery:
  seed_hosts: []
  seed_providers: file
http:
  publish_host: ${POD_NAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc
network:
  host: "0"
  publish_host: ${POD_IP}
node:
  attr:
    k8s_node_name: ${NODE_NAME}
  name: ${POD_NAME}
  store:
    allow_mmap: false
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
xpack:
  license:
    upload:
      types:
      - trial
      - enterprise
  security:
    authc:
      realms:
        file:
          file1:
            order: -100
        native:
          native1:
            order: -99
      reserved_realm:
        enabled: "false"
    enabled: "true"
    http:
      ssl:
        certificate: /usr/share/elasticsearch/config/http-certs/tls.crt
        certificate_authorities: /usr/share/elasticsearch/config/http-certs/ca.crt
        enabled: true
        key: /usr/share/elasticsearch/config/http-certs/tls.key
    transport:
      ssl:
        certificate: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.crt
        certificate_authorities:
        - /usr/share/elasticsearch/config/transport-certs/ca.crt
        - /usr/share/elasticsearch/config/transport-remote-certs/ca.crt
        enabled: "true"
        key: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.key
        verification_mode: certificate
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.