Master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster

Elastic Stack Elasticsearch
1

I'm trying to upgrade from 6.7.1 to 7.0.0 but getting the below error. I changed discovery.zen.ping.unicast.hosts to discovery.seed_hosts and added also cluster.initial_master_nodes pointing to the same master nodes. What am I missing for this to come up?

curl -XGET "localhost:9200/_cat/health?v&pretty"
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}

[WARN ][o.e.c.c.ClusterFormationFailureHelper] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes

2 Likes
2

Can you share the elasticsearch.yml file from one of your master nodes?

3

Also the full warning message from o.e.c.c.ClusterFormationFailureHelper (everything, from the timestamp at the start onwards) would be useful to see. You've truncated all the important bits out of your OP!

4

I have the same issue when migrating from 6.7.0 to 7.0.0

5

I had the same.
Put in the following into your elasticsearch.yml of your master node.
cluster.initial_master_nodes:
- node_name_or_ip

Start the first node and then the rest of the nodes.

3 Likes
6

I am running on kubernetes, node hostname or IP are not known in advance

7

Sorry then I wouldn't know how to solve this. This was on a regular host, no experience (yet) with Kubernetes.

8

In this situation you should use the node names, i.e. the values of node.name, not the hostnames or IP addresses.

1 Like
9

In case of kubernetes, the node.name is same as hostname. as it is name of the pod auto generated by kubernetes. Cannot be known in advance.

10

[2019-04-11T11:16:43,126][WARN ][o.e.d.SeedHostsResolver ] [elasticsearch-master-1-vl2l6] failed to resolve host [elasticsearch-master-site1.svc.cluster.local]

java.net.UnknownHostException: elasticsearch-master-site1-rta3.svc.cluster.local

at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_202]

at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_202]

at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_202]

at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-7.0.0.jar:7.0.0]

at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-7.0.0.jar:7.0.0]

at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:738) ~[elasticsearch-7.0.0.jar:7.0.0]

at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:143) ~[elasticsearch-7.0.0.jar:7.0.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.0.0.jar:7.0.0]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

11

While everything worked with version 6.7.0 with same setup

12

That's the default for node.name but you should be overriding it. See for instance how the Helm chart does it.

1 Like
13

I am doing it already to set es node.name to metadata.name of pod

14

Ok, so can you do what the Helm chart does to set cluster.initial_master_nodes too?

15

I cannot do it, as helm chart is using template helpers to extract that value, In my case i have simple statefulset defined. To do what helm chart is doing, i have to use helm, that means complete revamp of deployment.

16

If you are using a bare StatefulSet then AIUI the pod names are predictable:

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id

Is this not the case in your deployment?

17

I cannot rely on statefulset predictable behavior, I will rollback to ES 6.7.0 as this seems to be a major bug in ES 7.0.0

18

Can you clarify what you mean by this? Do you know of situations where it doesn't behave as documented?

19

Here is my elasticsearch.yml file

cluster.name: dblogging_es_dev
node.name: ${HOSTNAME}
node.master: true
node.data: false
node.ingest: false
node.max_local_storage_nodes: 1
path.data: /data/elasticsearch
path.repo: /dbbackup/d-gp2-es46-1
path.logs: /var/log/elasticsearch
network.host: site
network.bind_host: 0.0.0.0
transport.tcp.port: 9300
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: 5s
discovery.seed_hosts: ["d-gp2-es46-1", "d-gp2-es46-2", "d-gp2-es46-3"]
cluster.initial_master_nodes: ["d-gp2-es46-1", "d-gp2-es46-2", "d-gp2-es46-3"]
bootstrap.memory_lock: true

1 Like
20

Thanks @kyle_che, can you also share the whole o.e.c.c.ClusterFormationFailureHelper message?