Nodes fall out of the cluster es 7.9.1

Hello, we have the following problem: nodes can periodically fall out of the cluster

[2020-11-19T09:00:37,415][INFO ][o.e.c.s.MasterService    ] [h1-es03] node-left[{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{}{}{dimr} reason: followers check retry count exceeded], term: 87,
 version: 236727, delta: removed {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{}{}{dimr}}
[2020-11-19T09:00:39,763][INFO ][o.e.c.s.ClusterApplierService] [h1-es03] removed {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{}{}{dimr}}, term: 87, version: 236727, reason: Publication{term
=87, version=236727}
[2020-11-19T09:00:47,890][INFO ][o.e.c.s.MasterService    ] [h1-es03] node-join[{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{}{}{dimr} join existing leader], term: 87, version: 236730, delta:
 added {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{}{}{dimr}}
[2020-11-19T09:00:52,713][INFO ][o.e.c.s.ClusterApplierService] [h1-es03] added {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{}{}{dimr}}, term: 87, version: 236730, reason: Publication{term=8
7, version=236730}

3 nodes in a cluster, each can be a master and is a datanode. Elasticsearch 7.9.1 build opendistro from amazon.
In addition, we write to Elasticsearch using Apache Metron and when we try to write sometimes, we get errors: listener timeout after waiting for [30000] ms

Is that the node from the master or the node that dropped out?

FYI odfe is a fork by aws, so it might also be worth asking on their forum.

These messages are from the master, and the node-left reason indicates repeated health check failures, but +1 to Mark's FYI: the first thing you should do is to reproduce this with an official build to rule out that it's something that only affects the ODFE fork.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.