Master and data nodes have different cluster UUIDs

Elastic Stack Elasticsearch
1

Hi team,

We have a cluster hosted on AWS EC2 instances having
1 kibana (8GB root volume, non persistent)
3 master (8GB root volume, non-persistent)
12 data nodes (200GB persistent disk is attached per node)
We are using EC2 discovery plugin for discovering the cluster nodes. Due to some reasons, our master and kibana autoscaling groups recreated new instances and the older instances got replaced.

Current situation:
Master and Kibana nodes have new cluster UUIDS and the data nodes are running with older cluster UUID. Due to this, we are getting the following errors

2022-06-20T12:36:58,952][WARN ][o.e.c.c.Coordinator      ] [127.0.0.1-node1] failed to validate incoming join request from node [{127.0.0.1-node2}{ADsjli4JQzKtHRdmdp-bDA}{ZoGZPrbFScmb2xVutbI3cA}{10.193.20.56}{10.193.20.56:9300}{dilrt}{aws_availability_zone=eu-west-1a, ml.machine_memory=33252052992, ml.max_open_jobs=20, node_type=hot, xpack.installed=true, transform.node=true}]
org.elasticsearch.transport.RemoteTransportException: [127.0.0.1-node2][10.193.20.56:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid vJoCmAqBRmS_zw5Q1g02iQ than local cluster uuid lY1hx_21SVmxrdXAmuYElA, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:153) ~[elasticsearch-7.8.0.jar:7.8.0]
        at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:182) ~[?:?]
        at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:285) ~[?:?]
        at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:142) ~[?:?]
        at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:613) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

How can we make these masters join the old cluster? Any support would be highly apprecicated.

2

It looks like you're using Open Distro for Elasticsearch (ODFE) so you'll need to ask the ODFE folks for help. ODFE is quite different from the official Elasticsearch and the experts on ODFE don't answer questions here.

Alternatively, if you can reproduce the problem in Elasticsearch proper then we will try and help.

3

A couple of general notes:

Master eligible nodes MUST have persistent storage as they hold data that is critical for the operation of the cluster.

You may also need to look at this utility, although I am unsure whether this will or will not be impacted by OpenDistro being used. I would recommend asking at the OpenDistro forum.

4

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

5

Thank David and Christian for your suggestions. Will post it in open distro forum and try this node utility.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.