# Multiple masters elected during cluster crash - question about data consistency **URL:** \ **Category:** Elasticsearch\ **Created:** [November 28, 2012, 8:07am UTC](https://discuss.elastic.co/t/multiple-masters-elected-during-cluster-crash-question-about-data-consistency/9858 "2012-11-28T08:07:48Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Marek\_Skorek](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Marek\_Skorek](https://discuss.elastic.co/u/Marek_Skorek)\ **Post date:** [November 28, 2012, 8:07am UTC](https://discuss.elastic.co/t/multiple-masters-elected-during-cluster-crash-question-about-data-consistency/9858/1 "2012-11-28T08:07:48Z")
Hi, I met on OOME on all the three nodes in my cluster. After restarting each node from the system shell (concurrently) node 1 was started as the first one and was elected as a master one. Right after that the nodes 2 and 3 was started too but they could not see the 1st node had been started yet and the 2nd node was elected as master also. Then I saw that my single, three node cluster was splitted into two instances (one with a single node and the second one with to 2 left nodes). They started to work independently and two rivers was started concurrently on both "clusters". The question is: If two rivers are working concurrently is there any chance that after fixing the situation and "merge" broken clusters into a single one all the data will be available/indexed ? Or my river need to take care of the data consistency? Thanks for all of your advices 🙂 Regards, Marek. -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [November 29, 2012, 1:50am UTC](https://discuss.elastic.co/t/multiple-masters-elected-during-cluster-crash-question-about-data-consistency/9858/2 "2012-11-29T01:50:05Z")
It's possible that after "merge" you will end up with two shards that supposed to have the same, but contain different sets of data because they were parts of different clusters. It's somewhat of difficult situation to recover from. The best thing you can do in this case, is to remove one of the shards (by temporary setting number of replicas to 0, for example) and then reindex missing records. If you haven't done this already, I would recommend setting discovery.zen.minimum\_master\_nodes[http://www.elasticsearch.org/guide/reference/modules/discovery/zen.html](http://www.elasticsearch.org/guide/reference/modules/discovery/zen.html) to 2 (more than a half of master-eligible nodes in your cluster) in order to prevent such situation from happening in the future. On Wednesday, November 28, 2012 3:07:48 AM UTC-5, scoro wrote: > Hi, > > I met on OOME on all the three nodes in my cluster. > > After restarting each node from the system shell (concurrently) node 1 was > started as the first one and was elected as a master one. Right after that > the nodes 2 and 3 was started too but they could not see the 1st node had > been started yet and the 2nd node was elected as master also. > Then I saw that my single, three node cluster was splitted into two > instances (one with a single node and the second one with to 2 left nodes). > They started to work independently and two rivers was started concurrently > on both "clusters". > > The question is: > > If two rivers are working concurrently is there any chance that after > fixing the situation and "merge" broken clusters into a single one all the > data will be available/indexed ? Or my river need to take care of the data > consistency? > > Thanks for all of your advices 🙂 > > Regards, > Marek. -- ---
**Author:** ![Marek\_Skorek](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Marek\_Skorek](https://discuss.elastic.co/u/Marek_Skorek)\ **Post date:** [December 4, 2012, 3:21pm UTC](https://discuss.elastic.co/t/multiple-masters-elected-during-cluster-crash-question-about-data-consistency/9858/3 "2012-12-04T15:21:33Z")
Thank you Igor. That is the answer I was expected 🙂 W dniu czwartek, 29 listopada 2012 02:50:05 UTC+1 użytkownik Igor Motov napisał: > It's possible that after "merge" you will end up with two shards that > supposed to have the same, but contain different sets of data because they > were parts of different clusters. It's somewhat of difficult situation to > recover from. The best thing you can do in this case, is to remove one of > the shards (by temporary setting number of replicas to 0, for example) and > then reindex missing records. > > If you haven't done this already, I would recommend setting > discovery.zen.minimum\_master\_nodes[http://www.elasticsearch.org/guide/reference/modules/discovery/zen.html](http://www.elasticsearch.org/guide/reference/modules/discovery/zen.html) to > 2 (more than a half of master-eligible nodes in your cluster) in order to > prevent such situation from happening in the future. > > On Wednesday, November 28, 2012 3:07:48 AM UTC-5, scoro wrote: > > > Hi, > > > > I met on OOME on all the three nodes in my cluster. > > > > After restarting each node from the system shell (concurrently) node 1 > > was started as the first one and was elected as a master one. Right after > > that the nodes 2 and 3 was started too but they could not see the 1st node > > had been started yet and the 2nd node was elected as master also. > > Then I saw that my single, three node cluster was splitted into two > > instances (one with a single node and the second one with to 2 left nodes). > > They started to work independently and two rivers was started concurrently > > on both "clusters". > > > > The question is: > > > > If two rivers are working concurrently is there any chance that after > > fixing the situation and "merge" broken clusters into a single one all the > > data will be available/indexed ? Or my river need to take care of the data > > consistency? > > > > Thanks for all of your advices 🙂 > > > > Regards, > > Marek. -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/multiple-masters-elected-during-cluster-crash-question-about-data-consistency/9858/4 "2017-07-06T03:01:33Z")
Closed on July 6, 2017, 3:01am UTC