Running ELK on docker, Kibana says: Unable to retrieve version information from Elasticsearch nodes

Elastic Stack Kibana
1

I was referring to example given in the Elasticsearch documentation for starting elastic stack (elastic and kibana) on docker using docker compose. It gives example of docker compose version 2.2 file. So, I tried to convert it to docker compose version 3.8 file. Also, it creates three elastic nodes and has security enabled. I want to keep it minimal to start with. So I tried to turn off security and also reduce the number of elastic nodes to 2. This is how my current compose file looks like:

version: "3.8"

services:  
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-amd64
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    environment:
      - node.name=es01
      - cluster.name=docker-cluster
      - cluster.initial_master_nodes=es01
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
    deploy:
      resources:
        limits:
          memory: 1g 
    
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      # [
      #   "CMD-SHELL",
      #   # "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
      # ]

      # Changed to:
      test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]

      interval: 10s
      timeout: 10s
      retries: 120
  kibana:
    depends_on:
      - es01
    image: docker.elastic.co/kibana/kibana:8.0.0-amd64
    volumes:
      - kibanadata:/usr/share/kibana/data
    ports:
      - 5601:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://localhost:9200
    deploy:
      resources:
        limits:
          memory: 1g
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
volumes:
  esdata01:
    driver: local
  kibanadata:
    driver: local

Then, I tried to run it:

docker stack deploy -c docker-compose.nosec.noenv.yml elk
Creating network elk_default
Creating service elk_es01
Creating service elk_kibana

When I tried to check their status, it displayed following:

$ docker container list
CONTAINER ID   IMAGE                                                       COMMAND                  CREATED         STATUS                            PORTS                NAMES
3dcd08134e38   docker.elastic.co/kibana/kibana:8.0.0-amd64                 "/bin/tini -- /usr/l…"   3 minutes ago   Up 3 minutes (health: starting)   5601/tcp             elk_kibana.1.ng8aspz9krfnejfpsnqzl2sci
7b548a43c45c   docker.elastic.co/elasticsearch/elasticsearch:8.0.0-amd64   "/bin/tini -- /usr/l…"   3 minutes ago   Up 3 minutes (healthy)            9200/tcp, 9300/tcp   elk_es01.1.d9a107j6wkz42shti3n6kpfmx

I noticed that kibana's status gets stuck at (health: starting). When I checked Kibana's logs with command docker service logs -f elk_kibana, it had following WARN and ERROR lines:

[WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200

It seems that kibana is not able to connect with Elasticsearch, but why? Is it because of disabling of security and that we cannot have security disabled?

PS-1: Earlier, when I set Elasticsearch host as follows in kibana's environment in the docker compose file:

ELASTICSEARCH_HOSTS=https://es01:9200  # that is 'es01' instead of `localhost`

it gave me following error:

[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND es01

So, after checking this question, I changed es01 to localhost as specified earlier (that is in complete docker compose file content before PS-1.)

PS-2: Replacing localhost with 192.168.0.104 gives following error

[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 192.168.0.104:9200
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. write EPROTO 140274197346240:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
2

Is your Elasticsearch up and running? Something like this should print cluster details.

curl http://localhost:9200

If your ES instance is running then from the Kibana service you should configure it as

- ELASTICSEARCH_HOSTS=http://localhost:9200

Mind it's http , not https, since you removed all the security settings.

I'm not sure if this still would work as the stack is enabling security by default. If you study the documentation compose example you'll see that the configuration aims to have all traffic between nodes secured, but Kibana itself is exposed through http.

I just checked the 8.1 release from today and it works smoothly.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.