Elasticsearch - start checks failing as rootless containers

Elastic Stack Elasticsearch
,
1

Hi
I'm trying to get an Elasctsearch cluster running on RHEL9.4 with Podman.
My podman-compose files run and the cluster members try to start. but I get the following error message:

{"@timestamp":"2024-06-18T09:25:08.423Z", "log.level":"ERROR", "message":"node validation exception\n[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch. For more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.14/bootstrap-checks.html]\nbootstrap check failure [1] of [1]: memory locking requested for elasticsearch process but memory is not locked; for more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.14/_memory_lock_check.html]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"pdvmdev08"}

As described I have set the vm.max_map_count to 262144 and restarted all podman services. (I also restarted the server :wink: )
But I still got the same error on the elastic cluster members. (Elastic Version 8.14.1)

This is my podman/docker-compose file:

version: "3.8"

services:
 setup:
   image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
   volumes:
     - /data/certs:/usr/share/elasticsearch/config/certs
   user: "0"
   command: >
     bash -c '
       if [ x${ELASTIC_PASSWORD} == x ]; then
         echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
         exit 1;
       elif [ x${KIBANA_PASSWORD} == x ]; then
         echo "Set the KIBANA_PASSWORD environment variable in the .env file";
         exit 1;
       fi;
       if [ ! -f config/certs/ca.zip ]; then
         echo "Creating CA";
         bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
         unzip config/certs/ca.zip -d config/certs;
       fi;
       if [ ! -f config/certs/certs.zip ]; then
         echo "Creating certs";
         echo -ne \
         "instances:\n"\
         "  - name: es01\n"\
         "    dns:\n"\
         "      - es01\n"\
         "      - localhost\n"\
         "    ip:\n"\
         "      - 127.0.0.1\n"\
         "  - name: es02\n"\
         "    dns:\n"\
         "      - es02\n"\
         "      - localhost\n"\
         "    ip:\n"\
         "      - 127.0.0.1\n"\
         "  - name: es03\n"\
         "    dns:\n"\
         "      - es03\n"\
         "      - localhost\n"\
         "    ip:\n"\
         "      - 127.0.0.1\n"\
         "  - name: kibana\n"\
         "    dns:\n"\
         "      - kibana\n"\
         "      - localhost\n"\
         "    ip:\n"\
         "      - 127.0.0.1\n"\
         > config/certs/instances.yml;
         bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
         unzip config/certs/certs.zip -d config/certs;
       fi;
       echo "Setting file permissions"
       chown -R root:root config/certs;
       find . -type d -exec chmod 750 \{\} \;;
       find . -type f -exec chmod 640 \{\} \;;
       echo "Waiting for Elasticsearch availability";
       until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
       echo "Setting kibana_system password";
       until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
       echo "All done!";
     '
   healthcheck:
     test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
     interval: 1s
     timeout: 5s
     retries: 120

 es01:
   depends_on:
     setup:
       condition: service_healthy
   image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
   container_name: elasticsearch01
   labels:
     co.elastic.logs/module: elasticsearch
   volumes:
     - /data/certs:/usr/share/elasticsearch/config/certs
     - /data/elasticsearch/data/es01:/usr/share/elasticsearch/data
     - /data/elasticsearch/logs/es01:/usr/share/elasticsearch/logs
   ports:
     - 9201:9200
   environment:
     - node.name=es01
     - cluster.name=${CLUSTER_NAME}
     - cluster.initial_master_nodes=es01,es02,es03
     - discovery.seed_hosts=es02,es03
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
     - bootstrap.memory_lock=true
     - CLI_JAVA_OPTS=-Xms2g -Xmx2g
     - xpack.security.enabled=true
     - xpack.security.http.ssl.enabled=true
     - xpack.security.http.ssl.key=certs/es01/es01.key
     - xpack.security.http.ssl.certificate=certs/es01/es01.crt
     - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.enabled=true
     - xpack.security.transport.ssl.key=certs/es01/es01.key
     - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
     - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.verification_mode=certificate
     - xpack.license.self_generated.type=${LICENSE}
   mem_limit: ${ES_MEM_LIMIT}
   ulimits:
     memlock:
       soft: -1
       hard: -1
   healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
       ]
     interval: 10s
     timeout: 10s
     retries: 120
 
 es02:
   depends_on:
     es01:
       condition: service_healthy
   image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
   container_name: elasticsearch02
   labels:
     co.elastic.logs/module: elasticsearch
   volumes:
     - /data/certs:/usr/share/elasticsearch/config/certs
     - /data/elasticsearch/data/es02:/usr/share/elasticsearch/data
     - /data/elasticsearch/logs/es02:/usr/share/elasticsearch/logs
   ports:
     - 9202:9200
   environment:
     - node.name=es02
     - cluster.name=${CLUSTER_NAME}
     - cluster.initial_master_nodes=es01,es02,es03
     - discovery.seed_hosts=es01,es03
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
     - bootstrap.memory_lock=true
     - CLI_JAVA_OPTS=-Xms2g -Xmx2g
     - xpack.security.enabled=true
     - xpack.security.http.ssl.enabled=true
     - xpack.security.http.ssl.key=certs/es02/es02.key
     - xpack.security.http.ssl.certificate=certs/es02/es02.crt
     - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.enabled=true
     - xpack.security.transport.ssl.key=certs/es02/es02.key
     - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
     - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.verification_mode=certificate
     - xpack.license.self_generated.type=${LICENSE}
   mem_limit: ${ES_MEM_LIMIT}
   ulimits:
     memlock:
       soft: -1
       hard: -1
   healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
       ]
     interval: 10s
     timeout: 10s
     retries: 120

 es03:
   depends_on:
     es02:
       condition: service_healthy
   image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
   container_name: elasticsearch03
   labels:
     co.elastic.logs/module: elasticsearch
   volumes:
     - /data/certs:/usr/share/elasticsearch/config/certs
     - /data/elasticsearch/data/es03:/usr/share/elasticsearch/data
     - /data/elasticsearch/logs/es03:/usr/share/elasticsearch/logs
   ports:
     - 9203:9200
   environment:
     - node.name=es03
     - cluster.name=${CLUSTER_NAME}
     - cluster.initial_master_nodes=es01,es02,es03
     - discovery.seed_hosts=es01,es02
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
     - bootstrap.memory_lock=true
     - CLI_JAVA_OPTS=-Xms2g -Xmx2g
     - xpack.security.enabled=true
     - xpack.security.http.ssl.enabled=true
     - xpack.security.http.ssl.key=certs/es03/es03.key
     - xpack.security.http.ssl.certificate=certs/es03/es03.crt
     - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.enabled=true
     - xpack.security.transport.ssl.key=certs/es03/es03.key
     - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
     - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.verification_mode=certificate
     - xpack.license.self_generated.type=${LICENSE}
   mem_limit: ${ES_MEM_LIMIT}
   ulimits:
     memlock:
       soft: -1
       hard: -1
   healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
       ]
     interval: 10s
     timeout: 10s
     retries: 120

 kibana:
   depends_on:
     es01:
       condition: service_healthy
   image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
   container_name: kibana
   labels:
     co.elastic.logs/module: kibana
   volumes:
     - /data/certs:/usr/share/kibana/config/certs
     - /data/kibana/data:/usr/share/kibana/data
   ports:
     - ${KIBANA_PORT}:5601
   environment:
     - SERVERNAME=kibana
     - ELASTICSEARCH_HOSTS=https://es01:9200
     - ELASTICSEARCH_USERNAME=kibana_system
     - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
     - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
     - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
     - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
     - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
   mem_limit: ${KB_MEM_LIMIT}
   healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
       ]
     interval: 10s
     timeout: 10s
     retries: 120

 logstash:
   depends_on:
     es01:
       condition: service_healthy
     kibana:
       condition: service_healthy
   image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
   container_name: logtash
   labels:
     co.elastic.logs/module: logstash
   user: root
   volumes:
     - /data/logtash/data:/usr/share/logstash/data
     - /data/logtash/config/:/usr/share/logstash/pipeline/:ro
     - /data/certs:/usr/share/logstash/certs
   environment:
     - NODE_NAME="logstash"
     - xpack.monitoring.enabled=false
     - ELASTIC_USER=elastic
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
     - ELASTIC_HOSTS=https://es01:9200
   command: logstash -f /usr/share/logstash/pipeline/logstash.conf
   ports:
     - "5044:5044/udp"
   mem_limit: ${LS_MEM_LIMIT}

--> copied from the official Elastic repo.

Where is my error :frowning:
I cant find it?

Thanks for your help.

2

That doesn't look to be related to the error Elasticsearch is reporting:

[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch. For more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.14/bootstrap-checks.html] bootstrap check failure [1] of [1]: memory locking requested for elasticsearch process but memory is not locked; for more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.14/_memory_lock_check.html]

3

Hi David

You might be right. But the main problem is that it's not possible to run ELASTIC containers rootless with podman.
As soon as I started it as root all containers came up and worked as expected.

Does anyone have a solution for that issue?

4

I don't see the relationship between that and the message memory locking requested for elasticsearch process but memory is not locked. As described in the docs linked from that message, to address this you need to set bootstrap.memory_lock: false.

6

Hi David

:+1: :+1:
That helped and it worked now also in rootles.

THANKS for your help