# Master goes down, even after re-election, cluster is unresponsive **URL:** \ **Category:** Elasticsearch\ **Created:** [September 22, 2017, 6:05am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421 "2017-09-22T06:05:13Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [September 22, 2017, 6:05am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/1 "2017-09-22T06:05:13Z")
I have a 5 node cluster with 3 master eligible nodes and 2 dedicated data nodes. In the current cluster state, the current master node left owing to long GC's. After re-election, master was assigned to some other master eligible node. I get the following exception in my Elasticsearch logs on one of the dedicated slave node. ``` java.lang.IllegalStateException: cluster state from a different master than the current one, rejecting (received {cls-es-slave1}{Bh9_gR2jRqiLn3IjNfHYpA}{10.240.0.18}{10.240.0.18:9300}{master=true}, current {cls-es-master}{Z1O52E-fRIu2itHXL3l1Xg}{10.240.0.15}{10.240.0.15:9300}{master=true}) ``` Can anyone explain me whats going on? Thanks in advance. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 22, 2017, 6:06am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/2 "2017-09-22T06:06:25Z")
What version? ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [September 22, 2017, 6:15am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/3 "2017-09-22T06:15:03Z")
@warkolm Elasticsearch version: 2.4.5 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 22, 2017, 6:15am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/4 "2017-09-22T06:15:36Z")
Do you have minimum masters set? ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [September 22, 2017, 6:19am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/5 "2017-09-22T06:19:11Z")
No, But I guess when a new master is elected, the cluster should be healthy again automatically. If I had minimum master nodes set to 2 and I have 3 master eligible nodes out of which one is the current master. Now if the current master goes down and a new one is elected. The cluuster should be automatically up and healthy again. Am I missing something here? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 22, 2017, 6:22am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/6 "2017-09-22T06:22:52Z")
If you don't have min masters then it's possible that you had a split brain. ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [September 22, 2017, 6:29am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/7 "2017-09-22T06:29:05Z")
Even if I had set the minimum masters to 2, I would have faced this situation right? As a cluster state was being published from a different master than the current one as know to the dedicated data node. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 22, 2017, 6:38am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/8 "2017-09-22T06:38:00Z")
Well it looks like you have multiple masters sending out conflicting updates, whereas if you had min masters set then only 1 master would ever be active and there wouldn't be the conflict. ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [September 22, 2017, 6:39am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/9 "2017-09-22T06:39:42Z")
But if I had the min master set, then the cluster would have been inoperable as the cluster would have waited for those many masters to join. How is this fault tolerant? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 22, 2017, 6:44am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/10 "2017-09-22T06:44:08Z")
If you have 3 masters then min masters is 2, so you can still lose a master and maintain availability. If you don't set it then you risk data loss and corruption. It's a balance for sure, but I'd prefer consistency over availability myself, cause what's the point of having access to the data if it's wrong? ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [September 22, 2017, 7:24am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/11 "2017-09-22T07:24:03Z")
> [@Abhilash\_Bolla](#): > > java.lang.IllegalStateException: cluster state from a different master than the current one, rejecting (received {cls-es-slave1}{Bh9\_gR2jRqiLn3IjNfHYpA}{10.240.0.18}{10.240.0.18:9300}{master=true}, current {cls-es-master}{Z1O52E-fRIu2itHXL3l1Xg}{10.240.0.15}{10.240.0.15:9300}{master=true}) You mean the above exception might still have occurred even if I had min masters set? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 22, 2017, 7:31am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/12 "2017-09-22T07:31:21Z")
No. It's saying there that there are multiple masters, so setting min masters would have prevented that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 20, 2017, 7:31am UTC](https://discuss.elastic.co/t/master-goes-down-even-after-re-election-cluster-is-unresponsive/101421/13 "2017-10-20T07:31:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.