Data Node disconnects randomly from cluster

[Siva_Karan]

Hi Team,

Data node from the cluster some times automatically removed without any high heap usage.

with in sometime it joins automatically.

Please help me to sort it out.

Elasticsearch version :7.3.2
OS:ubunutu

2022-05-22T09:34:04,053][INFO ][o.e.c.s.MasterService ] [master1] node-left[{data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true} followers check retry count exceeded], term: 16, version: 385126, reason: removed {{data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true},}
[2022-05-22T09:34:04,237][INFO ][o.e.c.s.ClusterApplierService] [master1] removed {{data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true},}, term: 16, version: 385126, reason: Publication{term=16, version=385126}
[2022-05-22T09:34:04,255][INFO ][o.e.c.r.DelayedAllocationService] [master1] scheduling reroute for delayed shards in [59.7s] (21 delayed shards)
[2022-05-22T09:34:04,259][WARN ][o.e.c.r.a.AllocationService] [master1] [1][4] marking unavailable shards as stale: [0os98SECRnaYlfMr8DKTEA]
[2022-05-22T09:36:07,507][WARN ][o.e.c.r.a.AllocationService] [master1] [1][1] marking unavailable shards as stale: [yQj9BTm2TRKM8DfiOd5INw]
[2022-05-22T09:42:17,214][INFO ][o.e.c.s.MasterService ] [master1] node-join[{data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true} join existing leader, {data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true} join existing leader], term: 16, version: 385146, reason: added {{data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true},}
[2022-05-22T09:42:17,531][INFO ][o.e.c.s.ClusterApplierService] [master1] added {{data3}{N002utNPTYKE1G0NiuEmmQ}{SpuwRpCQSX6gkDvs8T1_MA}{ip}{ip:port}{d}{xpack.installed=true},}, term: 16, version: 385146, reason: Publication{term=16, version=385146}
[2022-05-22T09:51:07,253][WARN ][o.e.c.r.a.AllocationService] [master1] [1][2] marking unavailable shards as stale: [CpjBmzAMQSmLZEufTQIqVQ]
[2022-05-22T10:06:58,624][INFO ][o.e.c.r.a.AllocationService] [master1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[1][4]] ...]).

[DavidTurner]

@Siva_Karan

Elasticsearch version :7.3.2

This version went EOL well over a year ago, you should upgrade as a matter of urgency. Newer versions have significant improvements in stability and performance.

Newer versions also include documentation about diagnosing this kind of problem here in the reference manual. Some of that information applies to older versions too, but for the best troubleshooting experience you should upgrade.

[Siva_Karan]

Thanks for the information.

we have configured the clusters in GCP vm and Our seperate Datacenter.

our data center clusters are using same version,but they are stable cluster.

GCP clusters alone we are facing these kind of issue.