Master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes but ips look ok

Elastic Stack Elasticsearch
1

Hello,
I am trying to run elasticsearch node on 1 server using docker compose and then another node on different server.

My docker-compose file:

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
    container_name: es01
    environment:
      - node.name=es01
      - network.host=0.0.0.0
      - network.publish_host=serv1_ip
      - discovery.seed_hosts=serv2_ip
      - cluster.initial_master_nodes=serv1_ip,serv2_ip
      - cluster.name=docker-cluster
      - cluster.routing.allocation.enable=all
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local

networks:
  esnet:

Of course the second node on server2 is not running yet, but I'd at least expect this one to start correctly and then I would run similar docker-compose file on another server. Instead that's what I've got:

es01    | {"type": "server", "timestamp": "2019-08-08T11:43:36,744+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "starting ..."  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:43:37,165+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "publish_address {serv1_ip:9300}, bound_addresses {0.0.0.0:9300}"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:43:37,180+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:43:47,223+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [serv1_ip, serv2_ip] to bootstrap a cluster: have discovered [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [serv2_ip:9300] from hosts providers and [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:43:57,226+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [serv1_ip, serv2_ip] to bootstrap a cluster: have discovered [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [serv2_ip:9300] from hosts providers and [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:44:07,230+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [serv1_ip, serv2_ip] to bootstrap a cluster: have discovered [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [serv2_ip:9300] from hosts providers and [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:44:07,260+0000", "level": "WARN", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "timed out while waiting for initial discovery state - timeout: 30s"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:44:07,276+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "publish_address {serv1_ip:9200}, bound_addresses {0.0.0.0:9200}"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:44:07,277+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "started"  }
es01    | {"type": "server", "timestamp": "2019-08-08T11:44:17,233+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es01",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [serv1_ip, serv2_ip] to bootstrap a cluster: have discovered [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [serv2_ip:9300] from hosts providers and [{es01}{o7AceWa6SJ2H_-ZIT8CEmA}{bo1s5P9JSb-MW12tWcGRZw}{serv1_ip}{serv1_ip:9300}{dim}{ml.machine_memory=3973648384, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

Does anyone know what seems to be a problem?

2

cluster.initial_master_nodes must match the node name properties of your nodes. From the docs:

The initial master nodes should be identified by their node.name , which defaults to their hostname. Make sure that the value in cluster.initial_master_nodes matches the node.name exactly.

No, if you only have two master-eligible nodes then they are both required for the first election. If you want the cluster to form with just a single master-eligible node then you should set cluster.initial_master_nodes to just this single node's name, and leave it empty on the other node.

3

Well,
I don't know why, but somehow I thought that both node.name and ip_address can be used there. Also I tried using node names there, but probably I had some other parameters set with wrong values. I've spent on it so much time that I've almost lost hope :smile:

Anyway, I followed your tips and it is working as expected now.
Thank you so much for help.

I've one more question:
It is my understanding that if I set for example 3 nodes on 1 server and then 3 nodes on another server - everything connected in one cluster.

  1. Then if server 1 go down in case of some failure, then everything will still be working since server 2 can still serve all information ?
  2. cluster will stop working since splitting cluster on 2 servers will only help with performance issues but everything have to be working all the time?
4

Your cluster shouldn't be expected to work if half or more of the master-eligible nodes all fail at once. So if you have 6 master-eligible nodes you will need to keep at least 4 alive to keep the cluster working.

5

Ok, that makes sense.
But what is the correct way of creating master-eligible nodes on 2 different machines then? Cause it seems I can't just run one docker-compose file on machine1 and then another docker-compose file on machine2.
can it be done?

6

Of course, you would normally be running 3 matter nodes on 3 separate machines. I don't understand the question. Can you explain more clearly what doesn't work?

7

Ok, I will try to give more details:

As I pointed in my first post, I wanted to create one (for now) master node on machine 1 and then another master node on machine 2. To accomplish that I wanted to run one docker-compose file on machine 1 and then similar docker-compose file on machine 2. But since you replied that having only two master-eligible nodes they are both required at startup it seems that I can't do it like that.

When I follow your tips and provide cluster.initial_master_nodes with just one node on 1st machine and leave it empty on the 2nd one, the cluster starts just fine but my understanding is that in this case I have just 1 master node right?
and my question is:
is it possible to create 2 master nodes on 2 different machines using docker-compose like that? or maybe I have to take completely different approach?

8

No, there will be two master nodes if you do as you described. That said, there's no good reason to have two master nodes: it's better to have one master node if you don't need high availability, or three if you do.

9

Ah it is ok then, Creating 2 nodes on 2 different machines is just the first step for me.

Thank you so much for all the answers. Guess I know what I have to do.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.