# Nodes fall out of the cluster es 7.9.1 **URL:** \ **Category:** Elasticsearch\ **Created:** [November 23, 2020, 6:52am UTC](https://discuss.elastic.co/t/nodes-fall-out-of-the-cluster-es-7-9-1/256309 "2020-11-23T06:52:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![ThreatInter](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)\ **Post date:** [November 23, 2020, 6:52am UTC](https://discuss.elastic.co/t/nodes-fall-out-of-the-cluster-es-7-9-1/256309/1 "2020-11-23T06:52:35Z")
Hello, we have the following problem: nodes can periodically fall out of the cluster ```auto [2020-11-19T09:00:37,415][INFO][o.e.c.s.MasterService] [h1-es03] node-left[{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr} reason: followers check retry count exceeded], term: 87, version: 236727, delta: removed {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}} [2020-11-19T09:00:39,763][INFO][o.e.c.s.ClusterApplierService] [h1-es03] removed {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}}, term: 87, version: 236727, reason: Publication{term =87, version=236727} [2020-11-19T09:00:47,890][INFO][o.e.c.s.MasterService] [h1-es03] node-join[{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr} join existing leader], term: 87, version: 236730, delta: added {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}} [2020-11-19T09:00:52,713][INFO][o.e.c.s.ClusterApplierService] [h1-es03] added {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}}, term: 87, version: 236730, reason: Publication{term=8 7, version=236730} ``` 3 nodes in a cluster, each can be a master and is a datanode. Elasticsearch 7.9.1 build opendistro from amazon. In addition, we write to Elasticsearch using Apache Metron and when we try to write sometimes, we get errors: ```auto java.io.IOException: listener timeout after waiting for [30000] ms ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 23, 2020, 7:05am UTC](https://discuss.elastic.co/t/nodes-fall-out-of-the-cluster-es-7-9-1/256309/2 "2020-11-23T07:05:23Z")
Is that the node from the master or the node that dropped out? FYI odfe is a fork by aws, so it might also be worth asking on their forum. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 23, 2020, 9:25am UTC](https://discuss.elastic.co/t/nodes-fall-out-of-the-cluster-es-7-9-1/256309/3 "2020-11-23T09:25:58Z")
These messages are from the master, and the `node-left` reason indicates repeated health check failures, but +1 to Mark's FYI: the first thing you should do is to reproduce this with an official build to rule out that it's something that only affects the ODFE fork. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 21, 2020, 9:26am UTC](https://discuss.elastic.co/t/nodes-fall-out-of-the-cluster-es-7-9-1/256309/4 "2020-12-21T09:26:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.