New Node in Docker Implementation

Elastic Stack Elasticsearch
1

Hi, I am new here and I hope you can help me with a frustrating issue :frowning:

I have a Docker-based installation of the Elastic Stack. There were just two containers: es01 and kib01

Here the Docker-Compose yml File:

version: '2.2'
services:
  es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.11.1
container_name: es01
restart: always
environment:
  - node.name=es01
  - cluster.name=test-cluster
  - discovery.type=single-node
  - bootstrap.memory_lock=true
  - xpack.monitoring.collection.enabled=true
  - xpack.security.enabled=true
  - "ES_JAVA_OPTS=-Xms12288m -Xmx12288m"
ulimits:
  memlock:
    soft: -1
    hard: -1
volumes:
  - data01:/usr/share/elasticsearch/data
ports:
  - 9200:9200
networks:
  - elastic

  kib01:
image: docker.elastic.co/kibana/kibana:7.11.1
container_name: kib01
restart: always
env_file: .env
ports:
  - 5601:5601
environment:
  ELASTICSEARCH_URL: http://es01:9200
  ELASTICSEARCH_HOSTS: '["http://es01:9200"]'
  ELASTICSEARCH_USERNAME: kibana_system
  ELASTICSEARCH_PASSWORD: ${ES_PASS}
networks:
  - elastic

volumes:
  data01:
driver: local

networks:
  elastic:
driver: bridge

I wanted to add one or more nodes for testing. So I did a lot of research and updated the yml File:

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.11.1
    container_name: es01
    restart: always
    environment:
      - node.name=es01
      - cluster.name=test-cluster
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - xpack.monitoring.collection.enabled=true
      - xpack.security.enabled=true
      - "ES_JAVA_OPTS=-Xms12288m -Xmx12288m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.11.1
    container_name: es02
    restart: always
    environment:
      - node.name=es02
      - cluster.name=test-cluster
      - discovery.seed_hosts=es01
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - xpack.monitoring.collection.enabled=true    
      - xpack.security.enabled=true
      - "ES_JAVA_OPTS=-Xms12288m -Xmx12288m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data

    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.11.1
    container_name: kib01
    restart: always
    env_file: .env
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200"]'
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: ${ES_PASS}
      
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local

networks:
  elastic:
    driver: bridge

Unfortunately Kibana wont start and I just se the: Kibana server is not ready yet
The strange thing is in the logs there is no issue:

Bildschirmfoto 2021-05-31 um 18.21.02
Bildschirmfoto 2021-05-31 um 18.21.021015×130 103 KB

There is just this xpack error or warning but I don't think thats the problem.

I tried different things and updates the config file a hundred times an spent the whole day for troubleshooting.

I am looking forward for your ideas :slight_smile:
Thank you in advance!

EDIT

I just found more information in the logs:

kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","plugins","actions","actions"],"pid":7,"message":"APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","plugins","alerts","plugins","alerting"],"pid":7,"message":"APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":7,"message":"config sourced from: production cluster"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["error","elasticsearch","monitoring"],"pid":7,"message":"Request error, retrying\nGET http://es02:9200/_xpack?accept_enterprise=true => connect ECONNREFUSED 192.168.64.2:9200"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["error","elasticsearch","monitoring"],"pid":7,"message":"Request error, retrying\nGET http://es01:9200/_xpack?accept_enterprise=true => connect ECONNREFUSED 192.168.64.4:9200"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","elasticsearch","monitoring"],"pid":7,"message":"Unable to revive connection: http://es02:9200/"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"Unable to retrieve version information from Elasticsearch nodes."}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","elasticsearch","monitoring"],"pid":7,"message":"Unable to revive connection: http://es01:9200/"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","elasticsearch","monitoring"],"pid":7,"message":"No living connections"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:24+00:00","tags":["warning","plugins","monitoring","monitoring"],"pid":7,"message":"X-Pack Monitoring Cluster Alerts will not be available: No Living connections"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:54+00:00","tags":["warning","elasticsearch","monitoring"],"pid":7,"message":"Unable to revive connection: http://es02:9200/"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:54+00:00","tags":["warning","elasticsearch","monitoring"],"pid":7,"message":"Unable to revive connection: http://es01:9200/"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:54+00:00","tags":["warning","elasticsearch","monitoring"],"pid":7,"message":"No living connections"}
kib01    | {"type":"log","@timestamp":"2021-05-31T16:48:54+00:00","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

After that I compared my configs with other configs online and I tried to disable these two lines:

      - xpack.monitoring.collection.enabled=true
      - xpack.security.enabled=true

After that Kibana was ready and usable! yeey but Kibana didnt connect to the two nodes. After that I tried to enable the first line. After that Kibana was reachable and the connections to the two nodes were there.
So the big question is now why there is no issue with the first yml file with single-node but with two-nodes I got this issue. I think the problem is something with security and this x-pack-security parameter. Maybe I have to configure something a second time? But do I have to every time I add a Node?

Thank you all very much!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.