An election requires a node with id[-zQWNDqiRVe_EHGtFKi6Zw], have discovered [...] which is not a quorum

Elastic Stack Elasticsearch
1

Hi Everybody,

We use 2 have to nodes of elasticsearch on the cluster. One of them deleted by a human error. Now we are trying to recover the cluster from one elastic node and trying to add 2 news nodes.

Finally the cluster is never coming up . The error is like the master node has to aprove this but the master node was started from 0.(data folder empty). The error that we are getting is

master not discovered or elected yet, an election requires a node with id [-zQWNDqiRVe_EHGtFKi6Zw], have discovered [{elasticsearch1}{B5NU_B9NSkK6GUaQRVbpcw}{90DTUGvHR3GbJyRFAqGcBg}{10.0.52.247}{10.0.52.247:9300}{ml.machine_memory=19327352832, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum

The node has the data is in the "elasticsearch2".

The Setup is over docker swarm (below is the compose setup)


    services:
      elasticsearch1:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.1
        ports:
          - target: 9200
            published: 9200
            protocol: tcp
            mode: host
        environment:
            - "ES_JAVA_OPTS=-Xms10g -Xmx10g"
            - bootstrap.memory_lock=false
            - cluster.initial_master_nodes=elasticsearch2,elasticsearch3
            - discovery.seed_hosts=elasticsearch2,elasticsearch3
            - node.name=elasticsearch1
            - cluster.name=es-udsp-cluster
            - network.host=0.0.0.0
            - discovery.zen.ping_timeout=100s
            - discovery.zen.minimum_master_nodes=2
            - node.master=true
            - node.data=true
            - node.ingest=true
        volumes:
          - esmaster1:/usr/share/elasticsearch/data
        deploy:
          endpoint_mode: dnsrr
          mode: "replicated"
          replicas: 1
          resources:
            limits:
              memory: 11G


      elasticsearch2:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.1
        ports:
          - target: 9200
            published: 9202
            protocol: tcp
            mode: host
        environment:
            - "ES_JAVA_OPTS=-Xms10g -Xmx10g"
            - bootstrap.memory_lock=false
            - cluster.initial_master_nodes=elasticsearch2,elasticsearch3
            - discovery.seed_hosts=elasticsearch1,elasticsearch3
            - node.name=elasticsearch2
            - cluster.name=es-udsp-cluster
            - discovery.zen.ping_timeout=100s
            - discovery.zen.minimum_master_nodes=1
            - network.host=0.0.0.0
            - node.master=true
            - node.data=true
            - node.ingest=true
        volumes:
          - esmaster2:/usr/share/elasticsearch/data
        deploy:
          mode: "replicated"
          endpoint_mode: dnsrr
          replicas: 1
          resources:
            limits:
              memory: 11G

      elasticsearch3:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.1
        ports:
          - target: 9200
            published: 9203
            protocol: tcp
            mode: host
        environment:
            - "ES_JAVA_OPTS=-Xms10g -Xmx10g"
            - bootstrap.memory_lock=false
            - cluster.initial_master_nodes=elasticsearch2,elasticsearch3
            - discovery.seed_hosts=elasticsearch2,elasticsearch1
            - node.name=elasticsearch3
            - cluster.name=es-udsp-cluster
            - network.host=0.0.0.0
            - discovery.zen.ping_timeout=100s
            - discovery.zen.minimum_master_nodes=2
            - node.master=true
            - node.data=true
            - node.ingest=true
        volumes:
          - esmaster3:/usr/share/elasticsearch/data
        deploy:
          endpoint_mode: dnsrr
          mode: "replicated"
          replicas: 1
          resources:
            limits:
              memory: 11G

    volumes:
      postgres:
      esmaster1:
      esmaster2:
      esmaster3:

How can i solve this issue?

2

@DavidTurner

3

Since you've lost half of your nodes (i.e. one of the two) you cannot safely recover. You will need to restore your cluster from a recent snapshot. See https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html#high-availability-cluster-design-two-nodes for more details about building clusters that are resilient to failures.

1 Like
4

If i don't have snapshot because i have never setup them.

do i have not trick to recover the data?

Thanks @DavidTurner

I will use snapshot from now!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.