Elasticsearch 7.1.1 cluster not forming using discovery-ec2

I have setup 3 nodes in AWS using discovery-ec2 plugin but they are not forming cluster. They just formed as its own cluster.

elasticsearch.yml config on all three nodes

  • bootstrap.memory_lock: true
  • discovery.seed_providers: ec2
  • path.data: /mnt/elasticsearch/data
  • path.logs: /mnt/elasticsearch/log
  • cluster.name: clusterName
  • node.name: ${HOSTNAME}
  • network.host: _ ec2 _
  • cluster.initial_master_nodes: ${HOSTNAME}
  • cloud.node.auto_attributes: true
  • cluster.routing.allocation.awareness.attributes: aws_availability_zone
  • discovery.ec2.tag.role: master
  • discovery.ec2.tag.environment: Prod
  • discovery.ec2.groups: SecurityGroupID
  • discovery.ec2.any_group: false
  • discovery.ec2.host_type: private_ip
  • discovery.zen.minimum_master_nodes: 2

What could be wrong with the elasticsearch.yml config? I tried adding discovery.seed_hosts: pointing to 3 nodes IP but still not working.

I tried changing one of the nodes into data mode only and added all 3 nodes IP into discovery.seed_hosts: on 3 nodes.

Below is the logs on 2 master nodes
failed to validate incoming join request from node [{ip-10-24-3-70}{xBqemGsdTGiwAhWdcpyhXw}{7EHb9uUxTVGiG4-1FdtD1g}{10.24.3.70}{10.24.3.70:9300}{aws_availability_zone=eu-west-1a, ml.machine_memory=8264749056, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [ip-10-24-3-70][10.24.3.70:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _bTw3xAiSriOwjZ7r_HiPg than local cluster uuid awBETiqHS-2XTaiccHb50Q, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]

This looks wrong. From the docs:

You must set cluster.initial_master_nodes to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping and therefore to avoid the risk of data loss.

Since you have not set cluster.initial_master_nodes to the same list on each node, it sounds like you have formed multiple clusters.

Hi David,

Thanks for prompt reply.

Is there way to auto discover all cluster.initial_master_nodes?

All those nodes on AWS are launched with autoscaling and not possible to manually change all the time.

Not that we're aware of. The APIs that are available in EC2 don't give consistent enough answers to let us correctly do the kind of auto-discovery you are looking for. Note that you only need to set this setting for the initial cluster startup; once the cluster has formed, the setting is ignored and can be removed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.