Elastic master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes

Elastic Stack Elasticsearch
1

Hi , I have upgraded my elasticsearch cluster of 3 nodes from 7.4.0 to 7.5.1 and after completing the upgradation of the ES . I'm getting this error
master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes

here is my one of the node's configuration
cluster.name: JSR
node.name: elk-3
network.host: 192.168.1.72
http.port: 9200
transport.tcp.port: 9300
discovery.seed_hosts: ["192.168.1.59:9300", "192.168.1.62:9300", "192.168.1.72:9300"]
cluster.initial_master_nodes: ["192.168.1.59:9300","192.168.1.62:9300", "192.168.1.72:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.license.self_generated.type: basic
xpack.ml.enabled: false

Should i add any more to this configuration?

2

This node is brand-new and not the result of an upgrade.

3

Hi @DavidTurner

This node is brand-new and not the result of an upgrade
then how can i manage my cluster to elect an master node , before the upgradation it was working well , After the nodes been upgraded to 7.5.1 the problem started.

4

log [20:28:00.408] [info][licensing][plugins] Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: active

log [20:28:00.408] [info][licensing][plugins] Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: active
log [20:29:00.396] [warning][licensing][plugins] License information could not be obtained from Elasticsearch for the [data] cluster. TypeError: Cannot use 'in' operator to search for 'type' in null

@DavidTurner these are the kibana logs .... back to back it is importing and can't obtain the license from ES .

Two nodes have started and one node is still showing the error of the master not elected.
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}

This is one of the node's health , the other two are having health green

5

If two of the nodes report green health then the master has been elected, so the question is why this other node can't join it. You've only shared the first part of the log message, and the rest of the message is important.

6

[2020-01-08T14:11:19,019][INFO ][o.e.c.c.JoinHelper ] [elk-1] failed to join {elk-2}{6cWJvDm8TOaJIhtBQDR9FA}{5Givex09R32BuAOLB9ChGA}{192.168.1.62}{192.168.1.62:9300}{dilm}{mlmory=10317410304, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elk-1}{WSimmT65Rj26vOT4Jqsy2Q}{H4J-qqnvT3ecJwQXR9_tCw}{192.168.1.59}{192.168.1.59:9300}{distalled=true}, optionalJoin=Optional.empty}

org.elasticsearch.transport.RemoteTransportException: [elk-2][192.168.1.62:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node

at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:513) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:243) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.1.jar:7.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: org.elasticsearch.transport.RemoteTransportException: [elk-1][192.168.1.59:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid fmC6Hhd7TFOlYyCs8shEZw than locuuid qGkJy0s6Th-pFVTlW7yKFg, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.1.jar:7.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]

This is log for the first node which hasn't joined the cluster

7

failed to validate incoming join request from node [{elk-1}{WSimmT65Rj26vOT4Jqsy2Q}{IXD5y6UxQSyQZ8XGpCFCIQ}{192.168.1.59}{192.168.1.59:9300}{dilm}{ml.machine_memory=8203743232, ml.max_open_jobs=20, xpack.installed=true}]

org.elasticsearch.transport.RemoteTransportException: [elk-1][192.168.1.59:9300][internal:cluster/coordination/join/validate]Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid fmC6Hhd7TFOlYyCs8shEZw than locuuid qGkJy0s6Th-pFVTlW7yKFg, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.1.jar:7.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:834) [?:1.8.0_181]

[2020-01-08T15:23:07,800][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elk-2] adding template [.management-beats] for index patterns [.management-beats]

[2020-01-08T15:29:48,094][INFO ][o.e.c.m.MetaDataCreateIndexService] [elk-2] [.kibana_task_manager_3] creating index, cause [api], templates , shards [1]/[1], mappings [_doc]
[2020-01-08T15:29:48,607][INFO ][o.e.c.r.a.AllocationService] [elk-2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_task_manager_3][0]]

This is the log of the other node which is working fine and showing the cluster health as green.

image
image1256×560 33.5 KB

And check the image once please.

{
"name" : "elk-1",
"cluster_name" : "JSR",
"cluster_uuid" : "na",
"version" : {
"number" : "7.5.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
"build_date" : "2019-12-16T22:57:37.835892Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

And this is the response i got from the node which is not able to join the cluster and surprisingly it is not showing any UUID .
Could you please help me with this .

8

This node used to belong to a different cluster. It is not permitted to move nodes between clusters, as this can result in data loss.

9

That means , the node is lost from the cluster . I can't join them by anymeans ?
But previously those are the nodes in my cluster . But 1.59 has changed the UUID and now it is not generating the UUID also .
If it is a separate node , it should have some uuid for itself right?

10

I wouldn't say that the node is "lost from the cluster". It apparently never belonged to the same cluster as the other two nodes, because if they were in the same cluster then they would have the same cluster UUID.

Nodes have a node ID (e.g. WSimmT65Rj26vOT4Jqsy2Q in your cluster) but the cluster UUID identifies the whole cluster.

11

Thank you @DavidTurner .

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.