# Stale master elected in elastic version \< 7

**URL:** <https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882>\
**Category:** Elasticsearch\
**Created:** [May 3, 2022, 9:28pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882 "2022-05-03T21:28:12Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![johny1](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@johny1](https://discuss.elastic.co/u/johny1)\
**Post date:** [May 3, 2022, 9:28pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/1 "2022-05-03T21:28:12Z")

</div>

running version 6.x which relies on minimum master node config for quorum.  
the cluster has three master eligible nodes (1,2,3) and active master was (1). One master node (3) left the cluster due to an unknown error and rejoined when it was restarted manually after a few days. At this point of time, this master (3) had stale data and I presume it will gradually sync its internal state from active master (1). However, at this moment, the active master (1) left the cluster and (3) became the active master. This caused dangling handlers to shards created in past few days (while 3 was down) and lead to loss of data. My question is why did Elasticsearch elect (3) as active master while it had stale data and was in process of syncing from (1)? Does it provide any protection in this scenario and prefer (2) over (3) for master election?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 3, 2022, 10:51pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/2 "2022-05-03T22:51:57Z")

</div>

Welcome to our community! 😃  
First things, 6.X is [EOL](https://www.elastic.co/support/eol) and no longer supported, you should be looking to upgrade as a matter of urgency.

Can you post the output from the `_cluster/stats?pretty&human` API for us to look at? It'll help provide more context on your cluster.

---

<div class="post-metadata">

**Author:** ![johny1](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@johny1](https://discuss.elastic.co/u/johny1)\
**Post date:** [May 3, 2022, 11:03pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/3 "2022-05-03T23:03:33Z")

</div>

As its an internal cluster, I cannot post the details or output of the call. Here are some stats that might be of help. please let me know if you need specific details:

60 data nodes  
3 master eligible nodes  
minimum master nodes = 2  
400 indexes  
2600 shards replication 1:1  
16 billion documents  
70,000 segments

In general, what criteria Elasticsearch uses to elect an active master from 2 or 3 eligible nodes. Does staleness of a master state factor into this?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 4, 2022, 3:31am UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/4 "2022-05-04T03:31:30Z")

</div>

> [@johny1](#):
>
> Does it provide any protection in this scenario and prefer (2) over (3) for master election?

It does, but in V6 and earlier it's not watertight. You should upgrade to a version that isn't EOL as a matter of urgency.

---

<div class="post-metadata">

**Author:** ![johny1](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@johny1](https://discuss.elastic.co/u/johny1)\
**Post date:** [May 4, 2022, 12:23pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/5 "2022-05-04T12:23:26Z")

</div>

we are planning actively to upgrade soon to latest version. Do you have a reference on this topic? It will help us in improving our capacity planning against failure modes.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 4, 2022, 1:06pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/6 "2022-05-04T13:06:25Z")

</div>

What do you mean by a reference? The [EOL docs](https://www.elastic.co/support/eol) already linked in this thread show that you're using an unacceptably old version. You're missing out on literally years of bugfixes and performance improvements.

---

<div class="post-metadata">

**Author:** ![johny1](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@johny1](https://discuss.elastic.co/u/johny1)\
**Post date:** [May 4, 2022, 2:40pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/7 "2022-05-04T14:40:24Z")

</div>

My interest is in reference to latest docs documenting/explaining this behavior (around master election) in more detail that will help also explain why this scenario would have been avoided in current versions.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 4, 2022, 2:52pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/8 "2022-05-04T14:52:16Z")

</div>

[Discovery and cluster formation | Elasticsearch Guide [8.2] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html) perhaps?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2022, 2:52pm UTC](https://discuss.elastic.co/t/stale-master-elected-in-elastic-version-7/303882/9 "2022-06-01T14:52:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
