# ElasticSearch nodes fail at random

**URL:** <https://discuss.elastic.co/t/elasticsearch-nodes-fail-at-random/16515>\
**Category:** Elasticsearch\
**Created:** [March 21, 2014, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-fail-at-random/16515 "2014-03-21T07:32:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saurabh\_Minni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saurabh_minni/32/1103_2.png) [@Saurabh\_Minni](https://discuss.elastic.co/u/Saurabh_Minni)\
**Post date:** [March 21, 2014, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-fail-at-random/16515/1 "2014-03-21T07:32:04Z")

</div>

Hi,  
I am facing this weird issue where I have created an ES cluster with 3 data  
nodes and 1 master node.

All data nodes are dual octa core CPU with 32 GB RAM and the master is a  
quad core 16GB machine  
I am trying to insert around 3000 records per second with replica set as 1  
. Each record is 3KB in size.  
All goes well with the setup but suddenly one of the machine's load shoots  
up dramatically and then finally the machines becomes unreachable.  
It happens with 1 of the 3 data nodes at random.  
On further inspection I can see that page faults on that system for java  
process increases very quickly compared to others. System IO% also shoots  
up.

I have used mlockall and made refresh index rate as -1.

But inspite of all this, the cluster keeps on failing. I was hoping that  
insert rate with ElasticSearch would be really high reading at posts at  
various places.

Please let me know if you know of any setting that needs to be changed.

Thanks for your help in advance.  
Saurabh

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/22264709-ddd5-4389-ac6f-9640c7523036%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/22264709-ddd5-4389-ac6f-9640c7523036%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:41am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-fail-at-random/16515/2 "2017-07-06T01:41:41Z")

</div>