ES 1.4.2 Performance Issue

Elastic Stack Elasticsearch
1

Hi,

I have 8 nodes cluster running over Amazon AWS.
2 Master/Client nodes and 6 Data Nodes (each data node has 16 cores and 64GB of RAM - c4.4xlarge)
Elasticsearch version: is 1.4.2,
OS: Amazon AMI

In the last few days the CPU usage of the data nodes increased dramatically (from 20%-40% to 80%-100%) causing very slow searches.
Elasticsearch process is the one that consumes all of the CPU resources.

I performed the hot threads command as following:
http://:9200/_nodes/hot_threads

and here is the output:

Your help is much appreciated.

Thanks,
Hagai.

2

Hi,

I think, seeing your hot threads dump, may be the problem could be the merging segments. Take a look on this link. As far as are you performing some update every time? I mean, there are concurrently process running, like searching and bulk indexing when the CPU come up to 100%?

3

Also - we recommend 3 master nodes, otherwise you run a risk of split brain as you never have a majority quorum.