# Sorting out TLS for ELK, using Helm Chart, in AKS

**URL:** <https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security, docker\
**Created:** [October 6, 2021, 11:40pm UTC](https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080 "2021-10-06T23:40:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dcarlet](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@dcarlet](https://discuss.elastic.co/u/dcarlet)\
**Post date:** [October 6, 2021, 11:40pm UTC](https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080/1 "2021-10-06T23:40:56Z")

</div>

So, I am trying to set up the following configuration:

> 3 node elastic cluster  
> 1 node logstash  
> 1 node kibana

I am utilizing the Helm Charts ([GitHub - elastic/helm-charts: You know, for Kubernetes](https://github.com/elastic/helm-charts/)); and specifically I have referred to the "Security" examples for [ElasticSearch](https://github.com/elastic/helm-charts/tree/7.14/elasticsearch/examples/security/values.yaml) , [Kibana](https://github.com/elastic/helm-charts/blob/7.14/kibana/examples/security/values.yaml), and [Logstash.](https://github.com/elastic/helm-charts/blob/7.14/logstash/examples/security/values.yaml) I am utilizing my own TLS certs, as I want them to use existing TLS certificates I use for other things, such as TLS-enabled ingress for other services in the cluster. I am deploying into an AKS K8s cluster; and I am using a basic license.

However, when I have everything turned on, my Elastic cluster NEVER goes healthy. The logs make it seem like it should be happy, but it's not.

Here is my values.yaml file for Elasticsearch:

```auto
clusterName: "data-elk"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"

protocol: https
esJavaOpts: "-Xmx1g -Xms1g"

image: "<internal-registry>/docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.14.2"
imagePullSecrets:
  - name: intregcred

    
esConfig:
  elasticsearch.yml : |
    xpack.license.self_generated.type: basic

    xpack.security.enabled: true

    # Transport SSL
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
    xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
    xpack.security.transport.ssl.verification_mode: certificate

    # Http SSL
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
    xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
    #Disable the GeoIP Update (no license so it won't work)
    ingest.geoip.downloader.enabled: false

    #Turn on DEBUG logging for Discovery so we can figure out wtf
    logger.org.elasticsearch.discovery: DEBUG
    logger.org.elasticsearch.action: DEBUG
    logger.org.elasticsearch.broadcast: DEBUG
    logger.org.elasticsearch.cluster: DEBUG
    logger.org.elasticsearch.document: DEBUG
    logger.org.elasticsearch.env: DEBUG
    logger.org.elasticsearch.gateway: DEBUG
    logger.org.elasticsearch.recovery: DEBUG
    logger.org.elasticsearch.deprecation: ERROR

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.allow-http: "false"
  hosts:
    - host: elasticsearch.<domain>
      paths:
        - path: /
  tls:
    - secretName: self-wildcard-tls-cert
      hosts:
      - elasticsearch.<domain>

```

Here is my values file for Kibana:

```auto
elasticsearchHosts: "https://data-elk-master:9200"

image: "<internalreg>/docker.elastic.co/kibana/kibana"
imageTag: "7.14.2"
imagePullSecrets:
  - name: intregcred

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password

  - name: 'KIBANA_ENCRYPTION_KEY'
    valueFrom:
      secretKeyRef:
        name: kibana-credentials
        key: enckey

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/tls.key
      certificate: /usr/share/kibana/config/certs/tls.crt
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
      verificationMode: certificate
    telemetry.enabled: false
    logging.root.level: debug
        
protocol: https

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/kibana/config/certs

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.allow-http: "false"
  hosts:
    - host: kibana.<domain>
      paths:
        - path: /
  tls:
    - secretName: self-wildcard-tls-cert
      hosts:
      - kibana.<domain>

```

And here is my values.yaml for Logstash:

```auto
image: "<internalreg>.azurecr.us/docker.elastic.co/logstash/logstash"
imageTag: "7.14.2"
imagePullSecrets:
  - name: intregcred

persistence:
  enabled: true

logstashConfig:
  logstash.yml: |
    http.host: 0.0.0.0
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: '${ELASTICSEARCH_USERNAME}'
    xpack.monitoring.elasticsearch.password: '${ELASTICSEARCH_PASSWORD}'
    xpack.monitoring.elasticsearch.hosts: ["https://data-elk-master:9200"]
    xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certs/ca.crt
    
logstashPipeline:
  uptime.conf: |
    input { exec { command => "uptime" interval => 30 } }
    output { elasticsearch {
      hosts => ["https://data-elk-master:9200"]
      index => "logstash"
      user => '${ELASTICSEARCH_USERNAME}'
      password => '${ELASTICSEARCH_PASSWORD}'
      cacert => "/usr/share/logstash/config/certs/ca.crt"
      }
    }

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/logstash/config/certs

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.allow-http: "false"
  hosts:
    - host: logstash.<domain>
      paths:
        - path: /
          servicePort: 8080
  tls:
    - secretName: self-wildcard-tls-cert
      hosts:
      - logstash.<domain>

```

These configurations are otherwise good, as if I remove the xpack tls security stuff and change the protocol definition to http (and change the kibana/logstash stuff too), everything works great.

Interesting things I have noticed:

1. With the TLS config enabled, all the logs messages (will include the logs from the 3 nodes below) seem to be good, EXCEPT, that the leader will be repeatedly spitting out:

```auto
{"type": "server", "timestamp": "2021-10-06T22:48:05,409Z", "level": "DEBUG", "component": "o.e.c.s.MasterService", "cluster.name": "data-elk", "node.name": "data-elk-master-2", "message": "took [0s] to compute cluster state update for [maybe generate license for cluster]", "cluster.uuid": "RY55ngl2TBWr7cMIiCPH3w", "node.id": "3ixdfp4qT1CbQkutMIrE6A" }

```

and that message happens A LOT...like 109 times in 10 seconds a lot:

```auto
>>grep "maybe generate license for cluster" tlselastic-2-debug.log | wc -l
109

```

1. When I log into any of the cluster nodes and attempt to GET /\_license, I get:

```auto
[elasticsearch@data-elk-master-0 ~]$ curl -u admuser:<password> https://localhost:9200/_license
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

```

...if I do the same thing but go to http: here's what I get:

```auto
[elasticsearch@data-elk-master-0 ~]$ curl -u admuser:<password> http://localhost:9200/_license

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [admuser] for REST request [/_license]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [admuser] for REST request [/_license]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

```

admuser is the user I created in the elastic-credentials secret (ELASTIC\_USERNAME). Not sure why it can't authenticate? I'm assuming this is all related.

1. When I set protocol to HTTP in the values files and change the URLs in Kibana and Logstash, Elastic goes healthy, logstash works (and I can login to ES via the ingress, use the generated username/pw, and verify that stuff exists by going to /\_cat/indices), but Kibana does NOT.

I would attach logs, but apparently .txt files are not allowed? Only picture files?

---

<div class="post-metadata">

**Author:** ![dcarlet](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@dcarlet](https://discuss.elastic.co/u/dcarlet)\
**Post date:** [October 7, 2021, 1:25pm UTC](https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080/2 "2021-10-07T13:25:39Z")

</div>

Okay forgot about Pastebin;

Master 0 logs (not elected leader):

> **[Master-Elk-0 log - Pastebin.com](https://pastebin.com/0uDC0MPe)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Master 1 Logs (not elected leader):

> **[Master-Elk-1.log - Pastebin.com](https://pastebin.com/5kd1LxgL)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Master 2 Logs (elected leader, the long one):

> **[Master-Elk-2.log - Pastebin.com](https://pastebin.com/4WsmH9Bc)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

---

<div class="post-metadata">

**Author:** ![dcarlet](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@dcarlet](https://discuss.elastic.co/u/dcarlet)\
**Post date:** [October 25, 2021, 10:38pm UTC](https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080/3 "2021-10-25T22:38:53Z")

</div>

Anybody have any insights? Thoughts? Random "did you do x" type of comments?

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [October 25, 2021, 11:24pm UTC](https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080/4 "2021-10-25T23:24:55Z")

</div>

This doesn't really answer your question (I'm not familiar with how the Elasticsearch helm chart works), but have you looked into potentially using [Elastic Cloud on Kubernetes | ECK](https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html) instead? I've found fairly good success with this, and there seems to be far more documentation than with the helm charts.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2021, 11:25pm UTC](https://discuss.elastic.co/t/sorting-out-tls-for-elk-using-helm-chart-in-aks/286080/5 "2021-11-22T23:25:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.