After enabling xpack security readinessprobe fails

Elastic Stack Elasticsearch
1

I've got elk stack running through helm without security but now I was aked to turn it on.

So I generated certificates with elastic-certutil and put them in a secret and changed the config into

esConfig:
  elasticsearch.yml: |
    cluster.name: "docker-cluster"
    network.host: 0.0.0.0
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.license.self_generated.type: basic

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs
    defaultMode: 0755

Also changed protocol to https: protocol: https

Then helm upgrade and the elastics pod restarts without error. Yet it never becomes ready!

This is 100% because of readinessprobe not triggering, because this the readinessprobe in elasticsearch and that is NOT logged in the kubelet logs

                http () {
                  local path="${1}"
                  local args="${2}"
                  set -- -XGET -s

                  if [ "$args" != "" ]; then
                    set -- "$@" $args
                  fi

                  if [ -n "${ELASTIC_PASSWORD}" ]; then
                    set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"
                  fi

                  curl --connect-timeout 20 --output /dev/null -k "$@" "{{ .Values.protocol }}://127.0.0.1:{{ .Values.httpPort }}${path}"
}

What IS logged is the readinessprobe bash command from KIBANA and that fails all the time with a connect timeout.

I feel like Im in a chicken and egg situation here. How do I get xpack securitu going with kibana as frontend?

Here's the kubelet log showing the curl request from kibana with timeouts, this is with protocol http but https fails as well.

[bash -c #!/usr/bin/env bash -e
# Disable nss cache to avoid filling dentry cache when calling curl
# This is required with Kibana Docker using nss < 3.52
export NSS_SDB_USE_CACHE=no

http () {
local path="${1}"
            set -- -XGET -s --fail -L
  if [ -n "${ELASTICSEARCH_USERNAME}" ] && [ -n "${ELASTICSEARCH_PASSWORD}" ]; then
 set -- "$@" -u "${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD}"


STATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601${path}")
2

From Elastic Search to Elasticsearch

3

This helm chart requires that, if security is enabled, you provide the password for the elastic user to the container in the ELASTIC_PASSWORD environment variable.

                  if [ -n "${ELASTIC_PASSWORD}" ]; then
                    set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"
                  fi

Otherwise the curl health check will fail because it does not have valid credentials.

4

Ok thanks. I tried that by just hardcoding default credentials in the ready proby.
With xpack security disabled, protocol http the kubelet logs do get the script triggered.....and the pod becomes ready.

Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:           if [ "$args" != "" ]; then
Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:             set -- "$@" $args
Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:           fi
Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:
Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:           set -- "$@" -u "elastic:changeme"
Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:
Jun 19 08:17:56 LOGSTASH.nl kubelet[757]:           curl --output /dev/null -k "$@" .......

But with xpack.security enabled and protocol https the script is never logged in the kubelet logs, so it seems its not triggered at all.

6

found the solution

I enabled only transport level

xpack.security.transport.ssl.enabled: true

but http must also be enabled!

    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

then (of course) elastic runs on https and also the credentials must be provided

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.