One of our nodes is constantly leaving with "master not discovered yet"

Elastic Stack Elasticsearch
1

We have a cluster with 33 nodes, and one server (always the same one) is leaving the cluster quite often.

After restarting the service, it joins immediately.

The exact error is:

[2023-12-11T11:23:29,293][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elkdataw004.local] master not discovered yet: have discovered 
[
    {elkdataw004.local}{Hnd7niaLSoaqh12r01d0IA}{9b8EpeP4Q26XySf2I_j2nw}{elkdataw004.local}{elkdataw004.local}{x.x.159.183:9300}{sw}{8.11.1}{7000099-8500003},
    {elkmaster001.local}{05RymCqYR4SFoOC_gbJKGw}{i2NwqbM1TquDij8smGVITg}{elkmaster001.local}{elkmaster001.local}{x.x.4.16:9300}{m}{8.11.1}{7000099-8500003},
    {elkmaster002.local}{g9UxGx0nQuWH67uxDH0cgg}{DJlBkixiR_e4Qx2Y65wnFg}{elkmaster002.local}{elkmaster002.local}{x.x.4.17:9300}{m}{8.11.1}{7000099-8500003},
    {elkmaster003.local}{leNGcTahQDCQyebBqhTlKg}{MdN5wr9rQ9q-8iu2gjJGHA}{elkmaster003.local}{elkmaster003.local}{x.x.4.18:9300}{m}{8.11.1}{7000099-8500003}
];
discovery will continue using [
    x.x.4.17:9300, x.x.4.16:9300, x.x.4.18:9300
] from hosts providers and [
    {elkmaster002.local}{g9UxGx0nQuWH67uxDH0cgg}{DJlBkixiR_e4Qx2Y65wnFg}{elkmaster002.local}{elkmaster002.local}{x.x.4.17:9300}{m}{8.11.1}{7000099-8500003},
    {elkmaster003.local}{leNGcTahQDCQyebBqhTlKg}{MdN5wr9rQ9q-8iu2gjJGHA}{elkmaster003.local}{elkmaster003.local}{x.x.4.18:9300}{m}{8.11.1}{7000099-8500003},
    {elkmaster001.local}{05RymCqYR4SFoOC_gbJKGw}{i2NwqbM1TquDij8smGVITg}{elkmaster001.local}{elkmaster001.local}{x.x.4.16:9300}{m}{8.11.1}{7000099-8500003}
] from last-known cluster state;
node term 507, last-accepted version 18108627 in term 507;
joining [
    {elkmaster003.local}{leNGcTahQDCQyebBqhTlKg}{MdN5wr9rQ9q-8iu2gjJGHA}{elkmaster003.local}{elkmaster003.local}{x.x.4.18:9300}{m}{8.11.1}{7000099-8500003}
] in term [507] has status [waiting for local cluster applier] after [1.3m/79816ms];
for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html

How can I resolve this issue?

2

Did you try this?

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.