ELASTIC_PASSWORD and KIBANA_PASSWORD in docker-compose seems to be ignored?

Elastic Stack Elasticsearch
,
1

I am building out an automation project using ansible, docker and of course Elasticsearch. I have got SSL working, but I am really struggling with setting the passwords non-interactively.

My understanding from the docs, is that ELASTIC_PASSWORD should set the default elastic password, but all my requests come back as 'unable to authenticate user [elastic] for REST request'

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch.p12
xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
xpack.security.http.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch.p12
xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
xpack.security.transport.ssl.keystore.type: PKCS12

Dockerfile:

# Use the official Elasticsearch image as the base
FROM docker.elastic.co/elasticsearch/elasticsearch:8.5.1
WORKDIR /usr/share/elasticsearch/config/

COPY --chown=elasticsearch:elasticsearch elasticsearch.yml ./elasticsearch.yml
COPY --chown=elasticsearch:elasticsearch elasticsearch.keystore ./elasticsearch.keystore

ENV KEYSTORE_PASSWORD=123456789
ENV ELASTIC_PASSWORD=123456789
ENV KIBANA_PASSWORD=123456789

docker-compose.yml

services:
    elasticsearch:
        build:
          context: .
          dockerfile: Dockerfile
        container_name: elasticsearch
        environment:
        - discovery.type=single-node
        - xpack.security.enabled=true
        - xpack.security.enrollment.enabled=true
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        - KEYSTORE_PASSWORD=123456789
        - KIBANA_PASSWORD=123456789
        - ELASTIC_PASSWORD=123456789
        ports:
        - "9200:9200"
        - "9300:9300"
        networks:
        - elk
        hostname: "elasticsearch"
        volumes:
        - es_data:/usr/share/elasticsearch/data
        - /etc/elasticsearch/certs:/usr/share/elasticsearch/config/certs:ro
networks:
  elk:
    name: elk
    driver: bridge
volumes:
  es_data:
    driver: local

curl request:

sudo docker exec -it elasticsearch curl --cacert /usr/share/elasticsearch/config/certs/ca.crt  -X POST 'https://elastic:123456789@elasticsearch:9200/_security/user/kibana_system/_password' -d '{ "password": "123456789" }’

I also tried setting a password in the keystore called bootstrap.password and using that for the elastic user.

sudo docker exec -it elasticsearch curl --cacert /usr/share/elasticsearch/config/certs/ca.crt -u elastic:123456789 -X GET -H 'Content-Type: application/json' 'https://elasticsearch:9200'

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

It works when I set the password interactivity, but I am having issues automating that. It always hangs on 'Please confirm that you would like to continue [y/N]' and eventually times out. Similar issue with automatically resetting the password and trying to grab the result - the password doesn't seem to be in the output that ansible gets back.

- name: Reset Kibana password using expect
  ansible.builtin.expect:
    command: docker exec -it elasticsearch /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system --url https://elasticsearch:9200 -i
    responses:
      'Enter password for the elasticsearch keystore :': '{{ keystore_password }}'
      'Please confirm that you would like to continue [y/N]': 'y'
      'Enter password for [kibana_system]:': {{ elastic_kibana_password }}'
      'Re-enter password for [kibana_system]:': '{{ elastic_kibana_password }}'
    timeout: 30

Resetting the elastic password manually lets me use curl to reset Kibana, but that defeats the point of automation.

Sorry if I am misunderstanding something obvious, but I am struggling to see a solution. The KEYSTORE_PASSWORD must be working because ssl works.

2

Using ELASTIC_PASSWORD requires that the elasticsearch.keystore is writable, perhaps that's the issue you face.

In your case, since you're copying the keystore into the container, I'd suggest setting a password in that instead of using ELASTIC_PASSWORD

What you want to do (see Built-in users | Elasticsearch Guide [8.17] | Elastic) is
set bootstrap.password in your keystore file.

Assuming you have an ES install that you're using to build that keystore, you want

elasticsearch-keystore add bootstrap.password

Follow the prompts, and then use the resulting keystore file to bootstrap your nodes.

3

Thanks a lot for the reply.

I did try setting a bootstrap.password variable in my keystore, but that still didn't work as the elastic password, so I wasn't sure how it was supposed to work? The documentation on that seemed quite light. I attempted to use the curl command with the bootstrap.password , but still got unable to authenticate user [elastic].

This would be my preferred method as it is better than passing in an env var.

The elasticsearch.keystore file is owned by elasticsearch:elasticsearch permissions 600.

I did managed to automate setting the password using ansible, but adding it to the keystore would be a better approach if possible.

I noticed that the users file in the config folder is empty and is automatically changed to be owned by elasticsearch, so I wondered if something was going wrong there? I am not sure where the service account passwords are normally stored.

4

A note on terminology, "elastic" isn't really a service account (in the sense that Elasticsearch has a concept of service accounts, and elastic isn't one of them). Feel free to call it what makes sense to you, but you'll find docs that talk about service accounts, and elastic doesn't fit into that.

The password for elastic is stored in the .security system index. If you have a password there, then it will override the bootstrap.password.

Have you checked your logs? I think you've only supplied client side error messages, which by design tell you very little. The logs will tell you if there's a more serious problem going on.