ES, kibana both having ca.crt issues?

rik · August 17, 2026, 10:20pm

I'm not able to start up either the ES or kibana containers and I suspect the root cause has to do with ca-cert issues. i'm attaching my compose file below for reference

with ES, the log shows this error:

"@timestamp":"2026-08-17T21:38:47.019Z","log.level": "WARN","message":"caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.20.0.2:9200, remoteAddress=/192.168.65.1:58936}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#3]","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"95GZRlGZRlmu9m-yXVTy-g","elasticsearch.node.id":"D9rUdaBgR6aMHwW4AwADAA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster","error.type":"io.netty.handler.codec.DecoderException","error.message":"javax.net.ssl.SSLHandshakeException: (unknown_ca) Received fatal alert: unknown_ca","error.stack_trace":"io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: (unknown_ca) Received fatal alert: unknown_ca

via curl -k https://127.0.0.1:9200

	{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

with kibana the log shows

Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND elasticsearch

and curl -k https://127.0.0.1:5601 returns

	curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

my compose file is below. thanks very much for suggestions!

name: ocd4oak

volumes:
certs:
driver: local
node_modules:
driver: local
postgres_data:
driver: local
es01_data:
driver: local
kibana_data:
driver: local

networks:
default:
name: elasticsearch
external: false

services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ x${ES_PASSWORD} == x ]; then
echo "Set the ES_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne 
"instances:\n"
"  - name: es01\n"
"    dns:\n"
"      - es01\n"
"      - localhost\n"
"    ip:\n"
"      - 127.0.0.1\n"
"  - name: kibana\n"
"    dns:\n"
"      - kibana\n"
"      - localhost\n"
"    ip:\n"
"      - 127.0.0.1\n"
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 {} ;;
find . -type f -exec chmod 640 {} ;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ES_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{"password":"${KIBANA_PASSWORD}"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
interval: 1s
timeout: 5s
retries: 120

es01:
depends_on:
setup:
condition: service_healthy
image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
container_name: es01
ports:
- ${ES_PORT}:9200
volumes:
- certs:/usr/share/elasticsearch/config/certs
- es01_data:/usr/share/elasticsearch/data
environment:
- node.name=es01
- cluster.name=${ES_CLUSTER_NAME}
- discovery.type=single-node
- ELASTIC_PASSWORD=${ES_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${ES_LICENSE}
mem_limit: ${ES_MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt http://elasticsearch:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120

kibana:
depends_on:
es01:
condition: service_healthy
image: docker.elastic.co/kibana/kibana:${ES_VERSION}
container_name: ${KIBANA_CONTAINER_NAME}
volumes:
- certs:/usr/share/kibana/config/certs
- kibana_data:/usr/share/kibana/data
ports:
- 127.0.0.1:${KIBANA_PORT}:5601
extra_hosts:
- host.docker.internal:host-gateway
- model-runner.docker.internal:host-gateway
environment:
- SERVER_NAME=kibana
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
# - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
# - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
# - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://elasticsearch:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 30

RainTown · August 18, 2026, 2:11pm

Rather than me trying to troubleshoot your syntax, can I suggest starting from the official docs?

and/or

In latter case populating the .env file appropriately and maybe removing the es02/es03 sections/references if you only want a single node cluster for now.

Your file is similar, but with names a little bit different, and different quoting in places and ...

rik · August 18, 2026, 5:33pm

hi @RainTown , that's EXACTLY the source of my compose and .env files! i've done a few things like merge the setup and kibana services into the same file, changed some env variable names (STACK_VERSION --> ES_VERSION, LICENSE--> ES_LICENSE, etc) but tried to stay as close as possible. I can appreciate that you don't have time to help me debug the result, but moving forward from the docs is just what I'm trying to do.

EG, I just did a diff of the files (again) to look for things, and noticed that the doc's files used localhost vs elasticsearch in the health checks, and tried again; same errors.

stephenb · August 18, 2026, 8:33pm

hi @rik

Couple things.

During the debug cycle you have to completely delete the volume mounts for the cert, because if they were created wrong they will not be re-created. And thus will fail repeatedly
I am not sure exactly what you are copying but your health check is checking http not https

http plus several other places in yours

"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt http://elasticsearch:9200 | grep -q 'missing authentication credentials'",
]

I just worked from this from the example

This works comes up fine first time



services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      #- cluster.initial_master_nodes=es01,es02,es03
      #- discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

RainTown · August 19, 2026, 10:39am

Yeah, but respectfully, you started with a working config and somehow broke it, right? I also don't see what you are trying to do, nor do I really know your level of linux/docker/elasticsearch experience. The configs within the docs do work OOTB. If you start changing things, then change one thing at a time, and check you didn't break it at each and every step. Even things like quoting can make a difference, http vs https as was pointed out by @stephenb, cleaning up after failed attempts, etc.

Example, this is from your config:

if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne 
"instances:\n"
"  - name: es01\n"
"    dns:\n"
"      - es01\n"
"      - localhost\n"

This is from the docs/github config:

        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\

Your version is missing the trailing \ from every line, including the echo -ne . Is this maybe important?

Topic		Replies	Views
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. self signed certificate Kibana docker	21	356	March 27, 2025
Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate Kibana elastic-stack-security , docker	2	537	September 18, 2024
Security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes..version%2Cnodes..http.publish_address%2Cnodes.*.ip] Kibana elastic-stack-security , docker	1	581	November 20, 2023
Kibana with elastic form docker compose setup problem Kibana docker	5	1329	November 24, 2023
Trying to get my docker-compose to start Kibana with ssl Kibana docker	8	4370	March 2, 2022

ES, kibana both having ca.crt issues?

Related topics