# Every node for itself **URL:** \ **Category:** Elasticsearch\ **Created:** [August 29, 2012, 11:07am UTC](https://discuss.elastic.co/t/every-node-for-itself/8877 "2012-08-29T11:07:39Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Ozgur\_Orhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ozgur_orhan/32/2744_2.png) [@Ozgur\_Orhan](https://discuss.elastic.co/u/Ozgur_Orhan)\ **Post date:** [August 29, 2012, 11:07am UTC](https://discuss.elastic.co/t/every-node-for-itself/8877/1 "2012-08-29T11:07:39Z")
Hello, We have a problem that repeats itself every 5-12 hours period. When everything running smoothly (cluster is green) 1 node behaves irrational and every other node creates its own cluster (not 1/4 split, 1/1/1/1/1 split). This cluster mainly used for training so we have heavy traffic spikes on both reads and writes when jobs are triggered (also some continious small reads). 1. What happened to btrainer-1.138 ? 2. Even if 1 node (btrainer-1.138) behaves irrationally why didn't the cluster split by 1/4; why did other nodes lose the master btrainer-1.182 ? Setup : ``` 5 similar nodes : btrainer-1.182 (192.168.1.182) (Current Master before incident) btrainer-1.186 (192.168.1.186) btrainer-1.136 (192.168.1.136) btrainer-13.137 (192.168.13.137) btrainer-1.138 (192.168.1.138) ``` ES Configs : ``` cluster.name: btrainer discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [ "192.168.1.182:10300", ``` "192.168.1.186:10300", "192.168.1.136:10300", "192.168.13.137:10300", "192.168.1.138:10300" ] http.port: 10200 index.number\_of\_replicas: 4 transport.tcp.port: 10300 Java Options : ``` -Des-foreground=yes -Des.path.home=/elasticsearch -Xms4096m -Xmx20480m -Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -cp /elasticsearch/lib/*:/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.ElasticSearch ``` you can check the logs from the nodes : [https://gist.github.com/3510448](https://gist.github.com/3510448) Best Regards, Ozgur Orhan -- ---
**Author:** ![Ozgur\_Orhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ozgur_orhan/32/2744_2.png) [@Ozgur\_Orhan](https://discuss.elastic.co/u/Ozgur_Orhan)\ **Post date:** [August 29, 2012, 11:15am UTC](https://discuss.elastic.co/t/every-node-for-itself/8877/2 "2012-08-29T11:15:24Z")
Forgot to add, we are using version : 0.19.8 . On Wed, Aug 29, 2012 at 2:07 PM, Özgür Orhan [ozgurorhan@gmail.com](mailto:ozgurorhan@gmail.com) wrote: > Hello, > > We have a problem that repeats itself every 5-12 hours period. When > everything running smoothly (cluster is green) 1 node behaves > irrational and every other node creates its own cluster (not 1/4 > split, 1/1/1/1/1 split). This cluster mainly used for training so we > have heavy traffic spikes on both reads and writes when jobs are > triggered (also some continious small reads). > > 1. What happened to btrainer-1.138 ? > 2. Even if 1 node (btrainer-1.138) behaves irrationally why didn't the > cluster split by 1/4; why did other nodes lose the master > btrainer-1.182 ? > > Setup : > > ``` > 5 similar nodes : > > btrainer-1.182 (192.168.1.182) (Current Master before incident) > btrainer-1.186 (192.168.1.186) > btrainer-1.136 (192.168.1.136) > btrainer-13.137 (192.168.13.137) > btrainer-1.138 (192.168.1.138) > > ``` > > ES Configs : > > ``` > cluster.name: btrainer > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: [ "192.168.1.182:10300", > > ``` > > "192.168.1.186:10300", "192.168.1.136:10300", "192.168.13.137:10300", > "192.168.1.138:10300" ] > http.port: 10200 > index.number\_of\_replicas: 4 > transport.tcp.port: 10300 > > Java Options : > > ``` > -Des-foreground=yes > -Des.path.home=/elasticsearch > -Xms4096m > -Xmx20480m > -Djline.enabled=true > -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly > -cp /elasticsearch/lib/*:/elasticsearch/lib/sigar/* > org.elasticsearch.bootstrap.ElasticSearch > > ``` > > you can check the logs from the nodes : [https://gist.github.com/3510448](https://gist.github.com/3510448) > > Best Regards, > Ozgur Orhan -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:14am UTC](https://discuss.elastic.co/t/every-node-for-itself/8877/3 "2017-07-06T03:14:45Z")