# Kibana unable to authenticate user \[elastic\]

**URL:** <https://discuss.elastic.co/t/kibana-unable-to-authenticate-user-elastic/325678>\
**Category:** Kibana\
**Tags:** elastic-stack-security, docker\
**Created:** [February 16, 2023, 3:11am UTC](https://discuss.elastic.co/t/kibana-unable-to-authenticate-user-elastic/325678 "2023-02-16T03:11:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chromus](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@chromus](https://discuss.elastic.co/u/chromus)\
**Post date:** [February 16, 2023, 3:11am UTC](https://discuss.elastic.co/t/kibana-unable-to-authenticate-user-elastic/325678/1 "2023-02-16T03:11:39Z")

</div>

Hello Team.

I am running ELK using a docker multi-node deployment. My `docker-compose.yml` file is almost identical to the example one from the page ([Install Elasticsearch with Docker | Elasticsearch Guide [8.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/docker.html)), the only difference is that I am creating my own network and that I added `ingest.geoip.downloader.enabled=false`

If I take down `es02` and leave the other two nodes up (`es01` and `es03`) kibana works fine. But whenever I take down `es01` and leave up `es02` and `es03`, kibana stops working and causes the other two containers to fail

This is my `docker-compose.yml` file

```auto
version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    networks:
      - elk-network
    user: "0"
    command: >
      bash -c '
        if [x${ELASTIC_PASSWORD} == x]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [x${KIBANA_PASSWORD} == x]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [! -f config/certs/ca.zip]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [! -f config/certs/certs.zip]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          " - name: es01\n"\
          " dns:\n"\
          " - es01\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es02\n"\
          " dns:\n"\
          " - es02\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es03\n"\
          " dns:\n"\
          " - es03\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[-f config/certs/es01/es01.crt]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    networks:
      - elk-network
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - ingest.geoip.downloader.enabled=false
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    networks:
      - elk-network
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - ingest.geoip.downloader.enabled=false
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    networks:
      - elk-network
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata03:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - ingest.geoip.downloader.enabled=false
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    networks:
      -elk-network
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=["https://es01:9200"]
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local

networks:
  elk-network:
    external: true

```

Kibana logs:

```auto
[2023-02-16T03:01:43.762+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND es01
[2023-02-16T03:01:44.631+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2023-02-16T03:01:44.661+00:00][WARN][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND es01 error
[2023-02-16T03:01:44.778+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ConnectionError: getaddrinfo ENOTFOUND es01
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2023-02-16T03:01:44.780+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ConnectionError: getaddrinfo ENOTFOUND es01
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2023-02-16T03:01:44.784+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ConnectionError: getaddrinfo ENOTFOUND es01
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2023-02-16T03:01:46.074+00:00][WARN][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND es01 error
[2023-02-16T03:01:46.076+00:00][WARN][plugins.monitoring.monitoring] X-Pack Monitoring Cluster Alerts will not be available: getaddrinfo ENOTFOUND es01
[2023-02-16T03:01:50.992+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2023-02-16T03:01:51.015+00:00][WARN][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND es01 error
[2023-02-16T03:01:54.769+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2023-02-16T03:01:54.780+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ConnectionError: getaddrinfo ENOTFOUND es01
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)

```

If I update the hosts line for the kibana service in the `docker-compose.yml` into this `ELASTICSEARCH_HOSTS=["https://es01:9200","https://es02:9200","https://es03:9200"]` then I run into a different error in kibana:

```auto
[2023-02-16T02:55:52.119+00:00][INFO][plugins.security.routes] Logging in with provider "basic" (basic)
[2023-02-16T02:55:52.215+00:00][ERROR][plugins.security.user-profile] Failed to activate user profile: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for action [cluster:admin/xpack/security/profile/activate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for action [cluster:admin/xpack/security/profile/activate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}.
[2023-02-16T02:55:52.216+00:00][ERROR][http] ResponseError: security_exception: [security_exception] Reason: unable to authenticate user [elastic] for action [cluster:admin/xpack/security/profile/activate]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)
    at Security.activateUserProfile (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/security.js:60:16)
    at UserProfileService.activate (/usr/share/kibana/x-pack/plugins/security/server/user_profile/user_profile_service.js:111:26)
    at Authenticator.updateSessionValue (/usr/share/kibana/x-pack/plugins/security/server/authentication/authenticator.js:588:24)
    at Authenticator.login (/usr/share/kibana/x-pack/plugins/security/server/authentication/authenticator.js:196:37)
    at /usr/share/kibana/x-pack/plugins/security/server/routes/authentication/common.js:137:34
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:141:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:107:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

```

What am I doing wrong? Shouldn't I be able to have `es01` down and keep the cluster up and running?

I would appreciate any direction as to how could I achieve that.

Thanks in advance

---

<div class="post-metadata">

**Author:** ![chromus](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@chromus](https://discuss.elastic.co/u/chromus)\
**Post date:** [February 22, 2023, 10:46pm UTC](https://discuss.elastic.co/t/kibana-unable-to-authenticate-user-elastic/325678/2 "2023-02-22T22:46:03Z")

</div>

Found the solution myself thanks to:

> [@Authentication fails on a new node in a secure cluster](https://discuss.elastic.co/t/authentication-fails-on-a-new-node-in-a-secure-cluster/252517):
>
> Hello, I have an Elasticsearch cluster with 2 nodes, on which I enabled xpack security a while ago (TLS on both transport and HTTP + local authentication). I want to add a new node to the cluster, I configured the node with the transport and HTTP settings, I added the node in the discovery list on all the nodes. The node starts correctly but I cannot authenticate on it: curl https://SOC-V-DEV-ESNODE03:9200/\_xpack/security/\_authenticate?pretty -u elastic -k Enter host password for user 'elast…

I just needed to add the line:

```auto
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}

```

to node 2 and 3.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2023, 10:46pm UTC](https://discuss.elastic.co/t/kibana-unable-to-authenticate-user-elastic/325678/3 "2023-03-22T22:46:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.