# Split brains after long GCs **URL:** \ **Category:** Elasticsearch\ **Created:** [March 17, 2013, 12:54pm UTC](https://discuss.elastic.co/t/split-brains-after-long-gcs/11181 "2013-03-17T12:54:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![asher\_frenkel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asher_frenkel/32/23701_2.png) [@asher\_frenkel](https://discuss.elastic.co/u/asher_frenkel)\ **Post date:** [March 17, 2013, 12:54pm UTC](https://discuss.elastic.co/t/split-brains-after-long-gcs/11181/1 "2013-03-17T12:54:25Z")
Hi, I have a 6 nodes cluster running ES 0.20.5, the cluster currently have around 35M docs spread over 12 shards with 1 replica. each node has 48GB with 24GB Heap. I am experiencing at random time spike in heap usage followed by long GC. after the nodes finished the long GC cluster is getting into split brain situation with weird states where one of the cluster nodes is member of both sides of the split. minimum\_master\_nodes option does not help in this case since the node exists in two of the different cluster states. I appreciate any suggestion you can have to prevent this issues, especially the split brain since it causes corruption of our index. Thanks Asher -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Garth](https://avatars.discourse-cdn.com/v4/letter/g/ba9def/32.png) [@Garth](https://discuss.elastic.co/u/Garth)\ **Post date:** [March 17, 2013, 1:33pm UTC](https://discuss.elastic.co/t/split-brains-after-long-gcs/11181/2 "2013-03-17T13:33:46Z")
The minimum\_master\_nodes should be equal to the total number of nodes in the cluster. If you don't you can end up with multiple nodes as being masters. Also, you should look at changing the following values: Our configuration: discovery.zen.ping.timeout: 30s discovery.zen.fd.ping\_interval: 10s dicsovery.zen.fd.ping\_timeout: 30s discovery.zen.fd.ping\_retries: 480 When the master stops communicating and there is the beginnings of a split-brain, the entire cluster stops talking to the world. You cannot index/query etc. The default time is 1 minute. The above properties will allow you to extend that time out. In our configuration it's up to 80 minutes. This should never happen ( 80 minutes ) but if it does, you have a BIGGER problem going on. Also with this configuration if it were to get into 80 minutes, eventually all shards get put into a state where they reject all operations ( An exception I just do not recall ) on them and the entire cluster has to be rebooted. This also protects from have indexes that are out of whack and then trying to recover becomes hell. We had a split-brain where one node had 600k docs and another node had 6k. The mistake by the admin was to restart both nodes. Node that came up first wins. Guess which one won! Yep, 6k. We lost the 600k. On Mar 17, 2013 12:54 PM, "asher frenkel" [asher.frenkel@gmail.com](mailto:asher.frenkel@gmail.com) wrote: > Hi, > I have a 6 nodes cluster running ES 0.20.5, the cluster currently have > around 35M docs spread over 12 shards with 1 replica. > each node has 48GB with 24GB Heap. > > I am experiencing at random time spike in heap usage followed by long GC. > after the nodes finished the long GC cluster is getting into split brain > situation with weird states where one of the cluster nodes is member of > both sides of the split. > minimum\_master\_nodes option does not help in this case since the node > exists in two of the different cluster states. > > I appreciate any suggestion you can have to prevent this issues, > especially the split brain since it causes corruption of our index. > > Thanks > Asher > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 17, 2013, 1:48pm UTC](https://discuss.elastic.co/t/split-brains-after-long-gcs/11181/3 "2013-03-17T13:48:58Z")
First aid: try increase discovery.zen.ping.timeout: 60s (default: 3s) and the Zen fault detection discovery.zen.fd.ping\_timeout: 60s (default: 30s) discovery.zen.fd.ping\_interval: 60s (default: 1s) so the communication to the master node may take enough time to survive certain GC stalls. This is just a workaround - I don't know how long your GC stalls and how long a node can not respond. It is important to analyze these numbers. The default zen ping timout values are selected very carefully. They assume all nodes in the cluster are alive and can respond. But if you increase the timeout, you allow nodes may not respond, and that assumption will influence the cluster, long response times may be the consequence. It's just "sugar coating" the real problem. So increasing timeout is not a proper solution. Some hints to get closer to the cause: - check your code for the reason why your code can create "spikes" so GC must step in and run into JVM stall situations. There are situations where CMS GC can be improved, by avoiding edge cases, but it does not always work out. - if you must accept the "spikes" and large heaps and CMS GC can't be improved, try another GC algorithm which is optimized for large heaps and short stall times (G1 GC). Note, G1 GC is not default GC now in Java 7, and is not stable. G1 takes more CPU and decreases overall performance. It does not prevent spikes but it let the JVM respond within small time frames, the JVM is more reactive. - if all GC improvement strategies fail, consider a smaller heap per node - for example, more nodes and less heap size - so the "spikes" do not hurt so much Jörg Am 17.03.13 13:54, schrieb asher frenkel: > Hi, > I have a 6 nodes cluster running ES 0.20.5, the cluster currently have > around 35M docs spread over 12 shards with 1 replica. > each node has 48GB with 24GB Heap. > > I am experiencing at random time spike in heap usage followed by long GC. > after the nodes finished the long GC cluster is getting into split > brain situation with weird states where one of the cluster nodes is > member of both sides of the split. > minimum\_master\_nodes option does not help in this case since the node > exists in two of the different cluster states. > > I appreciate any suggestion you can have to prevent this issues, > especially the split brain since it causes corruption of our index. > > Thanks > Asher > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:46am UTC](https://discuss.elastic.co/t/split-brains-after-long-gcs/11181/4 "2017-07-06T02:46:08Z")