Elasticsearch 7.1.1 cluster not forming using discovery-ec2

Elastic Stack Elasticsearch
1

I have setup 3 nodes in AWS using discovery-ec2 plugin but they are not forming cluster. They just formed as its own cluster.

elasticsearch.yml config on all three nodes

  • bootstrap.memory_lock: true
  • discovery.seed_providers: ec2
  • path.data: /mnt/elasticsearch/data
  • path.logs: /mnt/elasticsearch/log
  • cluster.name: clusterName
  • node.name: ${HOSTNAME}
  • network.host: _ ec2 _
  • cluster.initial_master_nodes: ${HOSTNAME}
  • cloud.node.auto_attributes: true
  • cluster.routing.allocation.awareness.attributes: aws_availability_zone
  • discovery.ec2.tag.role: master
  • discovery.ec2.tag.environment: Prod
  • discovery.ec2.groups: SecurityGroupID
  • discovery.ec2.any_group: false
  • discovery.ec2.host_type: private_ip
  • discovery.zen.minimum_master_nodes: 2

What could be wrong with the elasticsearch.yml config? I tried adding discovery.seed_hosts: pointing to 3 nodes IP but still not working.

I tried changing one of the nodes into data mode only and added all 3 nodes IP into discovery.seed_hosts: on 3 nodes.

Below is the logs on 2 master nodes
failed to validate incoming join request from node [{ip-10-24-3-70}{xBqemGsdTGiwAhWdcpyhXw}{7EHb9uUxTVGiG4-1FdtD1g}{10.24.3.70}{10.24.3.70:9300}{aws_availability_zone=eu-west-1a, ml.machine_memory=8264749056, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [ip-10-24-3-70][10.24.3.70:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _bTw3xAiSriOwjZ7r_HiPg than local cluster uuid awBETiqHS-2XTaiccHb50Q, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]

2

This looks wrong. From the docs:

You must set cluster.initial_master_nodes to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping and therefore to avoid the risk of data loss.

Since you have not set cluster.initial_master_nodes to the same list on each node, it sounds like you have formed multiple clusters.

3

Hi David,

Thanks for prompt reply.

Is there way to auto discover all cluster.initial_master_nodes?

All those nodes on AWS are launched with autoscaling and not possible to manually change all the time.

4

Not that we're aware of. The APIs that are available in EC2 don't give consistent enough answers to let us correctly do the kind of auto-discovery you are looking for. Note that you only need to set this setting for the initial cluster startup; once the cluster has formed, the setting is ignored and can be removed.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.