Podman compose for kibana+elasticsearch+logstash

Elastic Stack Elasticsearch
,
1

I am trying to create a podman compose file to create instant Elasticsearch, Kibana, Logstash. All data should be backed up via volumes. The problem seems to be transferring the Kibana API key from Elastic to Kibana.
My test system is an AlmaLinux without SeLinux activated.

My current compose file looks like this, does anyone have any idea how I can get it to run correctly with podman?

version: "0.1"
services:
  elasticsearch:
    userns: "keep-id"
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - ELASTIC_PASSWORD=something_safe
    volumes:
      - ${HOME}/podman_project/elasticsearch/data:/usr/share/elasticsearch/data
      - ${HOME}/podman_project/elasticsearch/certs:/usr/share/elasticsearch/config/certs:ro
    ports:
      - "9200:9200"
    networks:
      - elastic_network
    healthcheck:
      test: curl -k -u elastic:$$ELASTIC_PASSWORD https://localhost:9200 && exit 0 || exit 1
      interval: 10s
      timeout: 10s
      retries: 20

  token-generator:
    userns: "keep-id"
    image: alpine:latest
    container_name: token-generator
    depends_on:
      - elasticsearch
    environment:
      - ELASTIC_PASSWORD=something_safe
    volumes:
      - ${HOME}/podman_project/kibana/config/token:/kibana-token
      - ${HOME}/podman_project/scripts/generate-token.sh:/generate-token.sh:ro
    networks:
      - elastic_network
    command: >
      sh -c "export ELASTIC_PASSWORD=something_safe && \
            apk update && apk add --no-cache curl jq && \
            sleep 10 && \
            RESPONSE=`curl -s -k -u elastic:$ELASTIC_PASSWORD -X POST -H 'Content-Type: application/json' https://elasticsearch:9200/_security/service/elastic/kibana/credential/token` && \
            echo 'Response: ' \$RESPONSE && \
            TOKEN=`echo \$RESPONSE | jq -r '.token.value'` && \
            echo 'Token: ' \$TOKEN && \
            echo \$TOKEN > /kibana-token/token && \
            sleep 5"
    restart: "no"

  kibana:
    userns: "keep-id"
    image: docker.elastic.co/kibana/kibana:8.10.2
    container_name: kibana
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      - ELASTICSEARCH_SERVICEACCOUNTTOKEN=file:/usr/share/kibana/config/token/token
      - ELASTICSEARCH_HOSTS=https://elasticsearch:9200
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=/usr/share/kibana/config/certs/privkey.pem
      - SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certs/fullchain.pem
      - ELASTICSEARCH_SSL_VERIFICATIONMODE=none
    volumes:
      - ${HOME}/podman_project/kibana/config:/usr/share/kibana/config
      - ${HOME}/podman_project/kibana/certs:/usr/share/kibana/config/certs:ro
      - ${HOME}/podman_project/kibana/config/token:/usr/share/kibana/config/token:ro
    ports:
      - "5601:5601"
    networks:
      - elastic_network
    command: >
      sh -c "echo 'Waiting for token file...'; \
            while [ ! -f /usr/share/kibana/config/token/token ]; do sleep 1; done; \
            echo 'Token found. Starting Kibana.'; \
            exec /usr/share/kibana/bin/kibana"

  logstash:
    userns: "keep-id"
    image: docker.elastic.co/logstash/logstash:8.10.2
    container_name: logstash
    environment:
      - LS_JAVA_OPTS=-Xms256m -Xmx256m
    volumes:
      - ${HOME}/podman_project/logstash/config:/usr/share/logstash/config:ro
      - ${HOME}/podman_project/logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ${HOME}/podman_project/logstash/input:/usr/share/logstash/input:ro
      - ${HOME}/podman_project/elasticsearch/certs:/usr/share/logstash/certs:ro
    ports:
      - "5000:5000"
    networks:
      - elastic_network

networks:
  elastic_network:
    driver: bridge
2

I'm not understanding few things here.

You are pre-populating the shares with certificates?, e.g.

Exactly how did you do that?

When I start with your podman file, those directories are obviously empty, and elasticsearch starts, but exits swiftly with

failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured PEM certificate_authorities [/usr/share/elasticsearch/config/certs/fullchain.pem] because the file does not exist

Also, and it may be the case that this works, but your kibana container is starting kibana after merely checking for the existence of the file /usr/share/kibana/config/token/token ? I thought that token value had to be explicitly added in kibana.yml ?

Mmm, does it? What makes you think that? Have you a some logs to show whats actually happening ... ?

3

Thanks, I'll try to reproduce my steps again. I first generated one myself as a certificate.
I tried to re-trace my steps and with these commands it should be done.

sudo setenforce 0

mkdir -v -p ~/podman_project/elasticsearch/data ~/podman_project/elasticsearch/certs ~/podman_project/kibana/certs ~/podman_project/kibana/config ~/podman_project/logstash/certs ~/podman_project/logstash/config
cd ~/podman_project/elasticsearch/certs
openssl req -x509 -newkey rsa:2048 -keyout privkey.pem -out fullchain.pem -days 365 -nodes -subj "/CN=localhost"
cp -v ~/podman_project/elasticsearch/certs/{privkey.pem,fullchain.pem} ~/podman_project/kibana/certs/
cp -v ~/podman_project/elasticsearch/certs/{privkey.pem,fullchain.pem} ~/podman_project/logstash/certs/

Thought it was not in the kibana.yml, never saw it in the yml file, so my intuition was rather use the variable “ELASTICSEARCH_SERVICEACCOUNTTOKEN”.
Can you tell me how I should implement it?

No, that was more of a guess.

I have now reset everything once and the error message from elastic:
failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable

It is probably still missing:

find ${HOME}/podman_project/ -type d -exec chmod 775 -v {} \;
find ${HOME}/podman_project/ -type f -exec chmod 664 -v {} \;

Whereas I thought this would have been taken care of with userns: “keep-id”.

The current status is that the Kibana container is waiting for the token file forever.

4

in

there is this setting

elasticsearch.serviceAccountToken
If your Elasticsearch is protected with basic authentication, this token provides the credentials that the Kibana server uses to perform maintenance on the Kibana index at startup. This setting is an alternative to elasticsearch.username and elasticsearch.password.
5

But so far I don't know how to implement this in podman, the problem remains the same. :thinking:

6

I don't know anything about podman, sorry.

But I do know if you script adding a line to your kibana.yml file with

elasticsearch.serviceAccountToken: "...put_token_here..."

it should mean kibana can start and connect to elasticsearch.

Similarly seems you are right with the env variable, as I added

ELASTICSEARCH_SERVICEACCOUNTTOKEN="...put_token_here..."

to my /etc/default/kibana (could be any file that is sourced before kibana starts), also removed entry from kibana,yml, and it still starts OK.

So one or other works for me.

I generated my token with:

curl -X POST -k -H "Authorization: Basic `echo -n elastic:mypassword|base64 -`" -H 'Content-Type: application/json' https://localhost:9200/_security/service/elastic/kibana/credential/token/service_account_token-kibana1?pretty=true

Note your script uses:

_security/service/elastic/kibana/credential/token

and I used

_security/service/elastic/kibana/credential/token/service_account_token-kibana1

which might be significant.

Good luck.

7

I can generate a token from the endpoint in my compose file, but Kibana is stuck with this authentication problem. When I use service_account_token-kibana1 the response is:

curl -X POST -k -H "Authorization: Basic `echo -n elastic:something_safe|base64 -`" -H 'Content-Type: application/json' https://localhost:9200/_security/service/elastic/kibana/credential/token/service_account_token-kibana1?pretty=true
{
  "error" : {
    "root_cause" : [
      {
        "type" : "version_conflict_engine_exception",
        "reason" : "[service_account_token-elastic/kibana/service_account_token-kibana1]: version conflict, document already exists (current version [1])",
        "index_uuid" : "LpK4QonITVGg5r2x2TQa2w",
        "shard" : "0",
        "index" : ".security-7"
      }
    ],
    "type" : "version_conflict_engine_exception",
    "reason" : "[service_account_token-elastic/kibana/service_account_token-kibana1]: version conflict, document already exists (current version [1])",
    "index_uuid" : "LpK4QonITVGg5r2x2TQa2w",
    "shard" : "0",
    "index" : ".security-7"
  },
  "status" : 409
}
8

Yeah, thats likely because one "elastic/kibana" token already exists at point you are running the command.

You can check with something like:

curl -u elastic:something_safe -k -s https://elasticsearch:9200/.security-7/_search | jq -cS '.hits.hits[]._source|select(.doc_type=="service_account_token")'

and reply will be something like

{"creation_time":1741996519501,"creator":{"email":null,"full_name":null,"metadata":{"_reserved":true},"principal":"elastic","realm":"reserved","realm_type":"reserved"},"doc_type":"service_account_token","enabled":true,"name":"kibanatoken","password":"{PBKDF2_STRETCH}10000$...random_stuff...=","username":"elastic/kibana","version":8170399}

Anyways ...

You have

Please check that file really has a valid token with curl. Can do that with:

curl -H "Authorization: Bearer $(cat /usr/share/kibana/config/token/token)" -k -s https://elasticsearch:9200

from inside the kibana container.

(Remember of course the tokens are stored in an elasticsearch index, and your config is persisting the elasticsearch data directory, so even if you keep stopping and starting the podman containers you need to cleanout the data directory in between tries)

    volumes:
      - ${HOME}/podman_project/elasticsearch/data:/usr/share/elasticsearch/data
9

In general terms of how elasticsearch+kibana security can be setup, and noting the official documentation is obviously the source of truth, I found this guide

to be excellently written and easy to follow, and I've used it myself successfully recently (8.17).

btw, I noticed you have

${HOME}/podman_project/scripts/generate-token.sh

mounted, but didn't share that file and its not clear what it does, maybe nothing?

Also

${HOME}/podman_project/kibana/config

is mounted, and will contain the kibana.yml which you also did not share?

also, try this for the token-generator

    command: >
      sh -c "export ELASTIC_PASSWORD=something_safe && \
            apk update && apk add --no-cache curl jq && \
            sleep 20 && \
            curl -s -k -u elastic:$$ELASTIC_PASSWORD -X POST -H 'Content-Type: application/json' https://elasticsearch:9200/_security/service/elastic/kibana/credential/token | jq -r '.token.value' | tee /kibana-token/token
            sleep 5"

I cant test if it works without the kibana.yml you are using.

10

Actually nothing important, the line with generate-token.sh is a remnant from another attempt where it was not yet integrated into the compose file.

Thank you Yes, there is also a corresponding key.
But token_* should work just as well as far as I understand. Here are several listed (maybe because of some tries):

curl -u elastic:something_safe -k -s https://localhost:9200/.security-7/_search?pretty
...
{"creation_time":1742035014536,"creator":{"email":null,"full_name":null,"metadata":{"_reserved":true},"principal":"elastic","realm":"reserved","realm_type":"reserved"},"doc_type":"service_account_token","enabled":true,"name":"service_account_token-kibana1","password":"{PBKDF2_STRETCH}10000$OXksFA6IlIGPekxBhvUZgYZASqp/bGgJ9Vw3Qpl1ySc=$Yk19a6Y61MqLf4dhxqKvQG6YaMH1uWnn16jOYnJRxNc=","username":"elastic/kibana","version":8100299}
{"creation_time":1742040969361,"creator":{"email":null,"full_name":null,"metadata":{"_reserved":true},"principal":"elastic","realm":"reserved","realm_type":"reserved"},"doc_type":"service_account_token","enabled":true,"name":"token_5Pa7mZUBqqyMx7KUsA6M","password":"{PBKDF2_STRETCH}10000$apuPvHV/qGruyEyeZ1yHhw3JPXE5p64LPM1NaWfvaTY=$UPS9ZZnbU2Bo0Qo2i3iNlzeWeR6I3kDDdF47SECSkhI=","username":"elastic/kibana","version":8100299}

The generated token also seems to work on the Kibana instance, but I noticed the token ID is missing in the list above from .security-7. Perhaps it means nothing.

podman exec -it kibana /bin/sh
$ env | grep ELASTICSEARCH
ELASTICSEARCH_SERVICEACCOUNTTOKEN=file:/usr/share/kibana/config/token/token
ELASTICSEARCH_SSL_VERIFICATIONMODE=none
ELASTICSEARCH_HOSTS=https://elasticsearch:9200

podman exec -it kibana /bin/sh
$ curl -k -H "Authorization: Bearer $(cat /usr/share/kibana/config/token/token)" https://elasticsearch:9200/_security/_authenticate?pretty
{
  "username" : "elastic/kibana",
  "roles" : [ ],
  "full_name" : "Service account - elastic/kibana",
  "email" : null,
  "token" : {
    "type" : "_service_account_index",
    "name" : "token_qunlnZUBhQjnMq4X96t7"
  },
  "metadata" : {
    "_elastic_service_account" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "_service_account",
    "type" : "_service_account"
  },
  "lookup_realm" : {
    "name" : "_service_account",
    "type" : "_service_account"
  },
  "authentication_type" : "token"
}


[kibana]          | {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-03-16T07:40:43.307+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception\n\tRoot causes:\n\t\tsecurity_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":2,"uptime":63.140619244},"trace":{"id":"ad59e84bb355daa3dc0b0418d2b47b41"},"transaction":{"id":"fd34ade2313b10ce"}}

[kibana]          | {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-03-16T07:40:27.427+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 10.89.0.27:9200","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":2,"uptime":47.260551083},"trace":{"id":"ad59e84bb355daa3dc0b0418d2b47b41"},"transaction":{"id":"fd34ade2313b10ce"}}

I'll look over it again later, maybe I can give another update by then.

11

OK, here's where I am ...

I can't test your Podman config as I dont have the kibana.yml file you are using, as I dont know whats in your ${HOME}/podman_project/kibana/config

But I do know that (with a completely blank canvas) your compose file didn't work for me as the token was not properly handled in the short-lived token-generator container and did not end up in the file /usr/share/kibana/config/token/token. This might have been because of a $ELASTIC_PASSWORD rather than $$ELASTIC_PASSWORD that is often needed here. I found that this

    command: >
      sh -c "export ELASTIC_PASSWORD=something_safe && \
            apk update && apk add --no-cache curl jq && \
            sleep 20 && \
            curl -s -k -u elastic:$$ELASTIC_PASSWORD -X POST -H 'Content-Type: application/json' https://elasticsearch:9200/_security/service/elastic/kibana/credential/token | jq -r '.token.value' | tee /kibana-token/token
            sleep 5"

populated the token file correctly. I also needed the sleep 20 as elasticsearch was not fully up after 10 seconds (you had sleep 10).

What you should do next depends on what (useful) data is in your elasticsearch indices? If "nothing" then I'd stop everything, replace your code with that code above, then

rm -rf $HOME/podman_project/elasticsearch/data/* $HOME/podman_project/kibana/config/token/token

and start the containers again. And share the kibana.yml file here if it still does not work.

If your elasticsearch indices already have stuff of value in them, then dont do that obviously.

12

I have now tested something again and added the corresponding tests/delays. maybe it has become too complex, but the error is still the same.
To make it easy to reproduce the state I have uploaded it here.

Thanks again. There is no data yet, it should first be a working template for the setup.
I'm sorry that the Kibana file was missing, but it doesn't have many more settings than the compose file. I hope with the repository it is now much easier to reproduce the steps.

13

OK, I got it to work.

First of all I start with a clean slate

$ rm -rf ~/podman_project/elasticsearch/data/*  ~/podman_project/kibana/config/token/token

So I am starting with just these files:

$ find ~/podman_project -type f
/home/kevin/podman_project/kibana/config/kibana-default.yml
/home/kevin/podman_project/kibana/certs/fullchain.pem
/home/kevin/podman_project/kibana/certs/privkey.pem
/home/kevin/podman_project/elasticsearch/certs/fullchain.pem
/home/kevin/podman_project/elasticsearch/certs/privkey.pem

Note I renamed kibana.yml to kibana-default.yml, and its contents are:

server.host: "0.0.0.0"
server.ssl.enabled: true
server.ssl.key: /usr/share/kibana/config/certs/privkey.pem
server.ssl.certificate: /usr/share/kibana/config/certs/fullchain.pem

elasticsearch.hosts: ["https://elasticsearch:9200"]
elasticsearch.serviceAccountToken: XXXX

elasticsearch.ssl.verificationMode: none

logging.appenders.default:
  type: console
  layout:
    type: json

Note the elasticsearch.serviceAccountToken: XXXX line.

When I do podman-compose up with config file below it goes as follows:

  • elasticsearch container starts
  • shortly afterwards Elasticsearch (the application) starts too
  • token-generator starts and after a bit puts the token in file /kibana-token/token
  • kibana container starts, waits for /usr/share/kibana/config/token/token to exist and be non-zero, then repalces the XXXX from kibana-default.yml with the real token from that file and creates a valid kibana.yml, then starts kibana (the application)

For me it all starts fine and I can login to kibana UI with elastic/something_safe.

Note if I down elasticsearch/kibana and up it again, without cleaning stuff out, it will still work but with a new (different/additional) token in kibana.yml. The old token is also still there, You can see this via:

curl -s -k -u elastic:something_safe -X GET -H 'Content-Type: application/json' https://localhost:9200/_security/service/elastic/kibana/credential | jq -S .tokens
{
  "count": 2,
  "nodes_credentials": {
    "_nodes": {
      "failed": 0,
      "successful": 1,
      "total": 1
    },
    "file_tokens": {}
  },
  "service_account": "elastic/kibana",
  "tokens": {
    "token_Gi7An5UBZ5ZGRHrlGTDL": {},
    "token_W2XBn5UBIoQdVLZp85pw": {}
  }
}

Note the 2 tokens.

The compose.yml file I used (I removed the logstash parts as that was not topic of this thread). Note I also tried with setting ELASTICSEARCH_SERVICEACCOUNTTOKEN, but could not get that to work in this scenario.

version: "0.1"
services:
  elasticsearch:
    userns: "keep-id"
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - ELASTIC_PASSWORD=something_safe
    volumes:
      - ${HOME}/podman_project/elasticsearch/data:/usr/share/elasticsearch/data
      - ${HOME}/podman_project/elasticsearch/certs:/usr/share/elasticsearch/config/certs:ro
    ports:
      - "9200:9200"
    networks:
      - elastic_network
    healthcheck:
      test: curl -k -u elastic:$$ELASTIC_PASSWORD https://localhost:9200 && exit 0 || exit 1
      interval: 10s
      timeout: 10s
      retries: 20

  token-generator:
    userns: "keep-id"
    image: alpine:latest
    container_name: token-generator
    depends_on:
      - elasticsearch
    environment:
      - ELASTIC_PASSWORD=something_safe
    volumes:
      - ${HOME}/podman_project/kibana/config:/kibana
      - ${HOME}/podman_project/kibana/config/token:/kibana-token
      - ${HOME}/podman_project/scripts/generate-token.sh:/generate-token.sh:ro
    networks:
      - elastic_network
    command: >
      sh -c "export ELASTIC_PASSWORD=something_safe && \
            apk update && apk add --no-cache curl jq && \
            sleep 20 && \
            echo SETTING_TOKEN && \
            curl -s -k -u elastic:$$ELASTIC_PASSWORD -X POST -H 'Content-Type: application/json' https://elasticsearch:9200/_security/service/elastic/kibana/credential/token | jq -r '.token.value' | tee /kibana-token/token && \
            echo -n TOKEN_has_been_set_to  && \
            cat /kibana-token/token && \
            sleep 5"
    restart: "no"

  kibana:
    userns: "keep-id"
    image: docker.elastic.co/kibana/kibana:8.10.2
    container_name: kibana
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      - ELASTICSEARCH_HOSTS=https://elasticsearch:9200
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=/usr/share/kibana/config/certs/privkey.pem
      - SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certs/fullchain.pem
      - ELASTICSEARCH_SSL_VERIFICATIONMODE=none
    volumes:
      - ${HOME}/podman_project/kibana/certs:/usr/share/kibana/config/certs:ro
      - ${HOME}/podman_project/kibana/config:/usr/share/kibana/config
      - ${HOME}/podman_project/kibana/config/token:/usr/share/kibana/config/token:ro
    ports:
      - "5601:5601"
    networks:
      - elastic_network
    command: >
      sh -c "echo 'Waiting for token file...'; \
            while [ ! -f /usr/share/kibana/config/token/token ]; do sleep 3 && echo waiting for file; done; \
            while [ ! -s /usr/share/kibana/config/token/token ]; do sleep 3 && echo waiting for file to be non-zero ; done; \
            echo 'Token found. Starting Kibana.'; \
            echo 'setting elasticsearch.serviceAccountToken to ' `cat /usr/share/kibana/config/token/token` ;\
            sed s/XXXX/`cat /usr/share/kibana/config/token/token`/ < /usr/share/kibana/config/kibana-default.yml > /usr/share/kibana/config/kibana.yml ;\
            cat /usr/share/kibana/config/kibana.yml ;\
            /usr/share/kibana/bin/kibana"

networks:
  elastic_network:
    driver: bridge
1 Like
14

I was somewhere on the wrong track by processing the token, thanks.

1 Like
15

To be honest I dont know if the environment variable method via this syntax ELASTICSEARCH_SERVICEACCOUNTTOKEN=file:/usr/share/kibana/config/token/token is valid. Nor setting elasticsearch.serviceAccountToken: /some/path/to/a/file/with/a/token in kibana.yml. If you got that from some documentation then please share that link as I'm curious now :slight_smile:

But amongst other issues there was the $ where a $$ was needed.

And good that you have success of course.