I have ELB stack running in kubernetes. But noticed issue with two out of 3 nodes with following error
{"type": "server", "timestamp": "2021-11-29T17:42:37,326Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "prod-eks-logs", "node.name": "es-master-2.elasticsearch", "message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [nnYFjBsuSN25apManWU-aA, nRV2B7R3QxWYiaXoxCMiAw, Bs1vhzBRQbyJVuv3jtWJbQ], have discovered [{es-master-2.elasticsearch}{nRV2B7R3QxWYiaXoxCMiAw}{HkGh-ZY3SmmyF-dwJaSw2g}{10.0.17.246}{10.0.17.246:9300}{cdfhilmrstw}, {es-master-0.elasticsearch}{Bs1vhzBRQbyJVuv3jtWJbQ}{RdXrp-17RrK1rQjk463wXw}{10.0.19.35}{10.0.19.35:9300}{cdfhilmrstw}, {es-master-1.elasticsearch}{nnYFjBsuSN25apManWU-aA}{9P45vPl3QBeqvIYkxxEIkw}{10.0.16.228}{10.0.16.228:9300}{cdfhilmrstw}] which is a quorum; discovery will continue using [10.0.19.35:9300, 10.0.16.228:9300] from hosts providers and [{es-master-2.elasticsearch}{nRV2B7R3QxWYiaXoxCMiAw}{HkGh-ZY3SmmyF-dwJaSw2g}{10.0.17.246}{10.0.17.246:9300}{cdfhilmrstw}] from last-known cluster state; node term 24, last-accepted version 500416 in term 24" }
Would like to know what is causing this and how can i prevent this in future.