We have a cluster with 33 nodes, and one server (always the same one) is leaving the cluster quite often.
After restarting the service, it joins immediately.
The exact error is:
[2023-12-11T11:23:29,293][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elkdataw004.local] master not discovered yet: have discovered
[
{elkdataw004.local}{Hnd7niaLSoaqh12r01d0IA}{9b8EpeP4Q26XySf2I_j2nw}{elkdataw004.local}{elkdataw004.local}{x.x.159.183:9300}{sw}{8.11.1}{7000099-8500003},
{elkmaster001.local}{05RymCqYR4SFoOC_gbJKGw}{i2NwqbM1TquDij8smGVITg}{elkmaster001.local}{elkmaster001.local}{x.x.4.16:9300}{m}{8.11.1}{7000099-8500003},
{elkmaster002.local}{g9UxGx0nQuWH67uxDH0cgg}{DJlBkixiR_e4Qx2Y65wnFg}{elkmaster002.local}{elkmaster002.local}{x.x.4.17:9300}{m}{8.11.1}{7000099-8500003},
{elkmaster003.local}{leNGcTahQDCQyebBqhTlKg}{MdN5wr9rQ9q-8iu2gjJGHA}{elkmaster003.local}{elkmaster003.local}{x.x.4.18:9300}{m}{8.11.1}{7000099-8500003}
];
discovery will continue using [
x.x.4.17:9300, x.x.4.16:9300, x.x.4.18:9300
] from hosts providers and [
{elkmaster002.local}{g9UxGx0nQuWH67uxDH0cgg}{DJlBkixiR_e4Qx2Y65wnFg}{elkmaster002.local}{elkmaster002.local}{x.x.4.17:9300}{m}{8.11.1}{7000099-8500003},
{elkmaster003.local}{leNGcTahQDCQyebBqhTlKg}{MdN5wr9rQ9q-8iu2gjJGHA}{elkmaster003.local}{elkmaster003.local}{x.x.4.18:9300}{m}{8.11.1}{7000099-8500003},
{elkmaster001.local}{05RymCqYR4SFoOC_gbJKGw}{i2NwqbM1TquDij8smGVITg}{elkmaster001.local}{elkmaster001.local}{x.x.4.16:9300}{m}{8.11.1}{7000099-8500003}
] from last-known cluster state;
node term 507, last-accepted version 18108627 in term 507;
joining [
{elkmaster003.local}{leNGcTahQDCQyebBqhTlKg}{MdN5wr9rQ9q-8iu2gjJGHA}{elkmaster003.local}{elkmaster003.local}{x.x.4.18:9300}{m}{8.11.1}{7000099-8500003}
] in term [507] has status [waiting for local cluster applier] after [1.3m/79816ms];
for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html
How can I resolve this issue?