High CPU load on some node

stephanustedy · March 5, 2018, 7:01am

Hi Guys,

I have face problem with our cluster for few days.
Suddenly the cluster spikes.

I have multiple heavy queries, which is :

aggregations
range queries

is there any suggestion how to improve or what to check?

ES version : 5.2.2
Shards : 12
Replica : 1
Docs : 51M
Size : ~90GB
Traffic: ~1500 queries / sec
Nodes : 14 node with setup :

11 data only node
1 data and master node
2 coordinator only node

hot_threads with options 100 thread :

gist.github.com

https://gist.github.com/stephanustedy/fbe8c0e5bc1c5f6e1f2d37b5fda37d33

gistfile1.txt

::: {product-172-21-43-28}{g1xa9YU9SMGOkZrX9SBe4Q}{HrO8DceMS5mFvVNWzTN9wg}{172.21.43.28}{172.21.43.28:9300}
   Hot threads at 2018-03-05T06:55:31.471Z, interval=500ms, busiestThreads=100, ignoreIdleThreads=true:
   
   87.6% (438ms out of 500ms) cpu usage by thread 'elasticsearch[product-172-21-43-28][search][T#13]'
     2/10 snapshots sharing following 37 elements
       java.lang.Class.getEnclosingMethod0(Native Method)
       java.lang.Class.getEnclosingMethodInfo(Class.java:1072)
       java.lang.Class.getEnclosingClass(Class.java:1272)
       java.lang.Class.getSimpleBinaryName(Class.java:1443)
       java.lang.Class.getSimpleName(Class.java:1309)

This file has been truncated. show original

Thanks!

system · April 2, 2018, 7:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.