Discovery-EC2 - master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered

Elastic Stack Elasticsearch
1

Version: 8.11.0, Build: rpm/d9ec3fa628c7b0ba3d25692e277ba26814820b20/2023-11-04T10:04:57.184859352Z, JVM: 21.0.1

[WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-10-10-10-1.us-west-2.compute.internal] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{ip-10-10-10-1.us-west-2.compute.internal}{M2ZZlhY9SgeiE6vp0MKrCA}{kgBcL3TnSfi4ixpOu-VGjA}{ip-10-10-10-1.us-west-2.compute.internal}{10.10.10.1}{10.10.10.1:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {ip-20-20-20-2.us-west-2.compute.internal}{eIBxZadpQvCK8_S7Ds9Ieg}{ddxtS_EHS3Wq5KjWzLMulg}{ip-20-20-20-2.us-west-2.compute.internal}{10.236.111.19}{10.236.111.19:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, 10.236.111.19:9300] from hosts providers and [{ip-10-10-10-1.us-west-2.compute.internal}{M2ZZlhY9SgeiE6vp0MKrCA}{kgBcL3TnSfi4ixpOu-VGjA}{ip-10-10-10-1.us-west-2.compute.internal}{10.10.10.1}{10.10.10.1:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html

After few seconds started to running the warning message:

[2023-12-16T16:25:08,175][WARN ][o.e.d.PeerFinder ] [ip-10-10-10-1.us-west-2.compute.internal] address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9301: Connection refused: /127.0.0.1:9301: Connection refused

2

Did you try that?

3

Hi @DavidTurner - no, I will try that link,

cluster.name: mypro-elasticsearch
discovery.seed_providers: ec2
discovery.ec2.availability_zones: us-west-2b,us-west-2c
discovery.ec2.tag.elasticsearch-discovery: Y
discovery.ec2.tag.Name: mypro-elasticsearch
discovery.ec2.host_type: private_ip
#
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
#
http.port: 9200
transport.port: 9300-9400
discovery.ec2.endpoint: ec2.us-west-2.amazonaws.com
network.host: 0.0.0.0
http.host: 0.0.0.0
bootstrap.memory_lock: true
http.max_content_length: 1000mb
indices.query.bool.max_clause_count: 10240
xpack.security.enabled: false
path.data: /var/lib/elasticsearch

# Path to log files:
path.logs: /var/log/elasticsearch

recent logs:

[2023-12-17T08:25:51,501][WARN ][o.e.d.PeerFinder] [ip-10-10-10-1.us-west-2.compute.internal] address [127.0.0.1:9300], node [null], requesting [false] discovery result: [ip-10-10-10-1.us-west-2.compute.internal][10.10.10.1:9300] successfully discovered local node {ip-10-10-10-1.us-west-2.compute.internal}{PYai92LaTXKc5vV7QjHn9A}{YcfnRPfcRZi8v3kPYcRy1Q}{ip-10-10-10-1.us-west-2.compute.internal}{10.10.10.1}{10.10.10.1:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003} at [127.0.0.1:9300]
[2023-12-17T08:25:52,367][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-10-10-10-1.us-west-2.compute.internal] 
master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] 
is empty on this node: have discovered [{ip-10-10-10-1.us-west-2.compute.internal}{PYai92LaTXKc5vV7QjHn9A}{YcfnRPfcRZi8v3kPYcRy1Q}
{ip-10-10-10-1.us-west-2.compute.internal}{10.10.10.1}{10.10.10.1:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}, 
{ip-10-10-10-2.us-west-2.compute.internal}{K_yqPmueShS09haXgJMH4Q}{BGDHfwVVT92bV7EXkWfFyA}{ip-10-10-10-2.us-west-2.compute.internal}
{10.10.10.2}{10.10.10.2:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}]; discovery will continue using 
[127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, **10.10.10.2:9300, 10.10.10.1:9300**]
 from hosts providers and [{ip-10-10-10-1.us-west-2.compute.internal}{PYai92LaTXKc5vV7QjHn9A}{YcfnRPfcRZi8v3kPYcRy1Q}
 {ip-10-10-10-1.us-west-2.compute.internal}{10.10.10.1}{10.10.10.1:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] 
 from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html
4

Hi Team, anyone can give the solutions for this?

5

Yes, use the troubleshooting docs mentioned previously.