ElasticSearch nodes fail at random

Hi,
I am facing this weird issue where I have created an ES cluster with 3 data
nodes and 1 master node.

All data nodes are dual octa core CPU with 32 GB RAM and the master is a
quad core 16GB machine
I am trying to insert around 3000 records per second with replica set as 1
. Each record is 3KB in size.
All goes well with the setup but suddenly one of the machine's load shoots
up dramatically and then finally the machines becomes unreachable.
It happens with 1 of the 3 data nodes at random.
On further inspection I can see that page faults on that system for java
process increases very quickly compared to others. System IO% also shoots
up.

I have used mlockall and made refresh index rate as -1.

But inspite of all this, the cluster keeps on failing. I was hoping that
insert rate with ElasticSearch would be really high reading at posts at
various places.

Please let me know if you know of any setting that needs to be changed.

Thanks for your help in advance.
Saurabh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/22264709-ddd5-4389-ac6f-9640c7523036%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.