Master not discovered or elected yet, after docker compose stop and start

Elastic Stack Elasticsearch
1

Hello,

Strange problem here (i think :slight_smile: )

I start elasticsearch on my laptop with docker-compose up -d es01, let the container start completly. Stop the containers with docker-compose stop. Then i do docker-compose up -d es01 again and elasticsearch gives me master not discovered or elected yet or when i check with

curl http://127.0.0.1:9200/_cat/health

i get:

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

When i stop the containers with docker=compose down -v
The containers will start normally after that

docker compose file is:

es01:
  image: docker.elastic.co/elasticsearch/elasticsearch:7.0.0
  container_name: es01
  volumes:
    - esdata01:/usr/share/elasticsearch/data
  environment:
  - node.name=es01
  - cluster.initial_master_nodes=es01,es02
  - cluster.name=docker-cluster
  - bootstrap.memory_lock=true
  - "ES_JAVA_OPTS=-Xms768m -Xmx768m"
  mem_limit: 1500m
  mem_reservation: 768m
  cpus: 0.70
  ulimits:
    memlock:
      soft: -1
      hard: -1
  ports:
    - "${ELASTICSEARCH_HOST_HTTP_PORT}:9200"
  depends_on:
    - es02
  #  - php-fpm
  networks:
    - esnet

es02:
  image: docker.elastic.co/elasticsearch/elasticsearch:7.0.0
  container_name: es02
  environment:
    - node.name=es02
    - discovery.seed_hosts=es01
    - cluster.initial_master_nodes=es01,es02
    - cluster.name=docker-cluster
    - bootstrap.memory_lock=true
    - "ES_JAVA_OPTS=-Xms768m -Xmx768m"
  mem_limit: 1500m
  mem_reservation: 768m
  cpus: 0.70
  ulimits:
    memlock:
      soft: -1
      hard: -1
  volumes:
    - esdata02:/usr/share/elasticsearch/data
  networks:
      - esnet

  esnet:

  volumes:
       esdata01: local
       esdata02: local
2

Can you share the Elasticsearch logs? Without them we can only speculate...

3

Hello,

You can see the log file here, hopefully it is enough information for you.
Best is you begin at the end of the file.

Thank you.

4

Perfect, thanks.

master not discovered or elected yet,
an election requires a node with id [alRw_LTeRqi8H9_0-0EAwg],
have discovered [] which is not a quorum;
discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
from hosts providers and [{es01}{6VQf4azMRd-DqmZorZ357A}{yvhISyJ3TtSRPeK_mQjtZg}{172.19.0.3}{172.19.0.3:9300}{ml.machine_memory=1572864000, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 1, last-accepted version 16 in term 1"

This node is looking for another node with ID alRw_LTeRqi8H9_0-0EAwg to perform an election. Looking earlier in the logs I see that node es02 has this ID. However this node can't find es02 at any of the listed addresses.

5

Ok that is clear. But how can i solve this :slight_smile:
Because i will start and stop the containers a lot.

6

I'd suggest starting node es02.

It looks like you only have a two-node cluster, so you'll need to start both nodes before the cluster will form.

7

What i did is stopping both containers. Then i start

docker-compose up -d es02

i let the container start and after that i start es01 with docker-compose. Same problem :frowning:

I shall provide logging of es02, maybe that is interesting for you.

Logging es02

8

Thanks, that's also useful. es02 says this about its network configuration:

publish_address {172.19.0.2:9300}, bound_addresses {0.0.0.0:9300}

However es01 says this:

discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]

It looks like es01 isn't looking for es02 in the right places. You need to make sure that each node knows how to find the other by setting discovery.seed_hosts.

9

Thank you very much! Problem solved :slight_smile:

1 Like
10

Just to make sure I understand the fix, you added an environment variable to es01 like:

  • discovery.seed_hosts=es02

I had the same problem, and doing this got things working again. If that is the right answer, than somebody should merge this change: PR: https://github.com/elastic/elasticsearch/pull/41394

Documentation that's missing the discovery.seed:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

1 Like
11

It is the right answer, yes. I didn't realise we were taking about the docker-compose file in the reference manual, but it does indeed look to be missing that. Thanks for pointing it out!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.