Sorting out TLS for ELK, using Helm Chart, in AKS

So, I am trying to set up the following configuration:

3 node elastic cluster
1 node logstash
1 node kibana

I am utilizing the Helm Charts (GitHub - elastic/helm-charts: You know, for Kubernetes); and specifically I have referred to the "Security" examples for ElasticSearch , Kibana, and Logstash. I am utilizing my own TLS certs, as I want them to use existing TLS certificates I use for other things, such as TLS-enabled ingress for other services in the cluster. I am deploying into an AKS K8s cluster; and I am using a basic license.

However, when I have everything turned on, my Elastic cluster NEVER goes healthy. The logs make it seem like it should be happy, but it's not.

Here is my values.yaml file for Elasticsearch:

clusterName: "data-elk"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"

protocol: https
esJavaOpts: "-Xmx1g -Xms1g"

image: "<internal-registry>/docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.14.2"
imagePullSecrets:
  - name: intregcred

    
esConfig:
  elasticsearch.yml : |
    xpack.license.self_generated.type: basic

    xpack.security.enabled: true

    # Transport SSL
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
    xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
    xpack.security.transport.ssl.verification_mode: certificate

    # Http SSL
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
    xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
    #Disable the GeoIP Update (no license so it won't work)
    ingest.geoip.downloader.enabled: false

    #Turn on DEBUG logging for Discovery so we can figure out wtf
    logger.org.elasticsearch.discovery: DEBUG
    logger.org.elasticsearch.action: DEBUG
    logger.org.elasticsearch.broadcast: DEBUG
    logger.org.elasticsearch.cluster: DEBUG
    logger.org.elasticsearch.document: DEBUG
    logger.org.elasticsearch.env: DEBUG
    logger.org.elasticsearch.gateway: DEBUG
    logger.org.elasticsearch.recovery: DEBUG
    logger.org.elasticsearch.deprecation: ERROR

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.allow-http: "false"
  hosts:
    - host: elasticsearch.<domain>
      paths:
        - path: /
  tls:
    - secretName: self-wildcard-tls-cert
      hosts:
      - elasticsearch.<domain>

Here is my values file for Kibana:

elasticsearchHosts: "https://data-elk-master:9200"

image: "<internalreg>/docker.elastic.co/kibana/kibana"
imageTag: "7.14.2"
imagePullSecrets:
  - name: intregcred

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password

  - name: 'KIBANA_ENCRYPTION_KEY'
    valueFrom:
      secretKeyRef:
        name: kibana-credentials
        key: enckey


kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/tls.key
      certificate: /usr/share/kibana/config/certs/tls.crt
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
      verificationMode: certificate
    telemetry.enabled: false
    logging.root.level: debug
        
protocol: https

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/kibana/config/certs

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.allow-http: "false"
  hosts:
    - host: kibana.<domain>
      paths:
        - path: /
  tls:
    - secretName: self-wildcard-tls-cert
      hosts:
      - kibana.<domain>

And here is my values.yaml for Logstash:

image: "<internalreg>.azurecr.us/docker.elastic.co/logstash/logstash"
imageTag: "7.14.2"
imagePullSecrets:
  - name: intregcred

persistence:
  enabled: true

logstashConfig:
  logstash.yml: |
    http.host: 0.0.0.0
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: '${ELASTICSEARCH_USERNAME}'
    xpack.monitoring.elasticsearch.password: '${ELASTICSEARCH_PASSWORD}'
    xpack.monitoring.elasticsearch.hosts: ["https://data-elk-master:9200"]
    xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certs/ca.crt
    
logstashPipeline:
  uptime.conf: |
    input { exec { command => "uptime" interval => 30 } }
    output { elasticsearch {
      hosts => ["https://data-elk-master:9200"]
      index => "logstash"
      user => '${ELASTICSEARCH_USERNAME}'
      password => '${ELASTICSEARCH_PASSWORD}'
      cacert => "/usr/share/logstash/config/certs/ca.crt"
      }
    }

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/logstash/config/certs

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.allow-http: "false"
  hosts:
    - host: logstash.<domain>
      paths:
        - path: /
          servicePort: 8080
  tls:
    - secretName: self-wildcard-tls-cert
      hosts:
      - logstash.<domain>

These configurations are otherwise good, as if I remove the xpack tls security stuff and change the protocol definition to http (and change the kibana/logstash stuff too), everything works great.

Interesting things I have noticed:

  1. With the TLS config enabled, all the logs messages (will include the logs from the 3 nodes below) seem to be good, EXCEPT, that the leader will be repeatedly spitting out:
{"type": "server", "timestamp": "2021-10-06T22:48:05,409Z", "level": "DEBUG", "component": "o.e.c.s.MasterService", "cluster.name": "data-elk", "node.name": "data-elk-master-2", "message": "took [0s] to compute cluster state update for [maybe generate license for cluster]", "cluster.uuid": "RY55ngl2TBWr7cMIiCPH3w", "node.id": "3ixdfp4qT1CbQkutMIrE6A"  }

and that message happens A LOT...like 109 times in 10 seconds a lot:

>>grep "maybe generate license for cluster" tlselastic-2-debug.log | wc -l
109
  1. When I log into any of the cluster nodes and attempt to GET /_license, I get:
[elasticsearch@data-elk-master-0 ~]$ curl -u admuser:<password> https://localhost:9200/_license
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

...if I do the same thing but go to http: here's what I get:

[elasticsearch@data-elk-master-0 ~]$ curl -u admuser:<password> http://localhost:9200/_license

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [admuser] for REST request [/_license]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [admuser] for REST request [/_license]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

admuser is the user I created in the elastic-credentials secret (ELASTIC_USERNAME). Not sure why it can't authenticate? I'm assuming this is all related.

  1. When I set protocol to HTTP in the values files and change the URLs in Kibana and Logstash, Elastic goes healthy, logstash works (and I can login to ES via the ingress, use the generated username/pw, and verify that stuff exists by going to /_cat/indices), but Kibana does NOT.

I would attach logs, but apparently .txt files are not allowed? Only picture files?

Okay forgot about Pastebin;

Master 0 logs (not elected leader):

Master 1 Logs (not elected leader):

Master 2 Logs (elected leader, the long one):