ES 1.4.2 Performance Issue

Hagai_T · September 8, 2015, 10:15am

Hi,

I have 8 nodes cluster running over Amazon AWS.
2 Master/Client nodes and 6 Data Nodes (each data node has 16 cores and 64GB of RAM - c4.4xlarge)
Elasticsearch version: is 1.4.2,
OS: Amazon AMI

In the last few days the CPU usage of the data nodes increased dramatically (from 20%-40% to 80%-100%) causing very slow searches.
Elasticsearch process is the one that consumes all of the CPU resources.

I performed the hot threads command as following:
http://:9200/_nodes/hot_threads

and here is the output:

github.com

hagait/es/blob/master/hot-threads-dump.txt

::: [Sergei Kravinoff][u4U7PtikRym7Vx3wTUZhuw][ip-10-0-110-62][inet[/10.0.110.62:9300]]{data=false, client=true}
   
    5.2% (25.8ms out of 500ms) cpu usage by thread 'pool-5-thread-13'
     10/10 snapshots sharing following 8 elements
       sun.misc.Unsafe.park(Native Method)
       java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
   
    2.1% (10.3ms out of 500ms) cpu usage by thread 'http-apr-8080-Poller'
     10/10 snapshots sharing following 3 elements
       org.apache.tomcat.jni.Poll.poll(Native Method)
       org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1767)
       java.lang.Thread.run(Thread.java:745)
   
    1.1% (5.2ms out of 500ms) cpu usage by thread 'http-apr-8080-exec-70'

This file has been truncated. show original

Your help is much appreciated.

Thanks,
Hagai.

lzerma · September 9, 2015, 1:42pm

Hi,

I think, seeing your hot threads dump, may be the problem could be the merging segments. Take a look on this link. As far as are you performing some update every time? I mean, there are concurrently process running, like searching and bulk indexing when the CPU come up to 100%?

warkolm · September 10, 2015, 5:58am

Also - we recommend 3 master nodes, otherwise you run a risk of split brain as you never have a majority quorum.