I have 8 nodes cluster running over Amazon AWS.
2 Master/Client nodes and 6 Data Nodes (each data node has 16 cores and 64GB of RAM - c4.4xlarge)
Elasticsearch version: is 1.4.2,
OS: Amazon AMI
In the last few days the CPU usage of the data nodes increased dramatically (from 20%-40% to 80%-100%) causing very slow searches.
Elasticsearch process is the one that consumes all of the CPU resources.
I performed the hot threads command as following:
I think, seeing your hot threads dump, may be the problem could be the merging segments. Take a look on this link. As far as are you performing some update every time? I mean, there are concurrently process running, like searching and bulk indexing when the CPU come up to 100%?