High CPU consumption

Eugene_Strokin · July 10, 2015, 7:02am

Hello,
I'm experiencing very high CPU usage, nearly 100%. The documentation says that ES is light on CPU. I assume that it could be some heavy queries. But how can I prove it and find what queries are eating CPU?
I've ran hot_threads:

gist.github.com

https://gist.github.com/strokine/dc209dacbebf958fd0be

gistfile1.txt

::: [es-3][dLcodqFYSWmlzMhMWN9u3g][inet[/10.128.1.236:9300]]
   
   76.8% (383.8ms out of 500ms) cpu usage by thread 'elasticsearch[es-3][cache][T#16]'
     8/10 snapshots sharing following 13 elements
       sun.nio.ch.NativeThread.current(Native Method)
       sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
       sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:670)
       org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162)
       org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:270)
       org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)

This file has been truncated. show original

But I don't understand what could be the problem here?
Please help,
Thanks,
Eugene

warkolm · July 10, 2015, 7:38am

Looks like you may have an overloaded cluster, can you add another node?

Eugene_Strokin · July 10, 2015, 11:25am

Thanks, looks like it is the case, I've added another node, CPU usage dropped.
Out of curiosity, did you get to this conclusion based on the hot_threads info? If so, what exactly shows the overload?
Thanks,
Eugene

Harlin_ES · July 10, 2015, 4:08pm

Are any of the queries coming in sorted queries or aggregations?

Eugene_Strokin · July 10, 2015, 4:10pm

We have a lot of sorted queries. This is the business needs. No aggregations though, as far as I can see.

Harlin_ES · July 10, 2015, 4:21pm

Sorted queries can cause issues with Elasticsearch if the cluster isn't prepared for them. (getting individual fields to sort on out of an inverted index is hard). ES has better ways of getting this field data. If you aren't already using one of these approaches I would high suggest either eagerly loading field data or using doc values

haochun · May 20, 2017, 6:29am

@Eugene_Strokin i have encountered the same probole,did you find the cause?i want to find which query,but i have no way:sob:

Eugene_Strokin · May 29, 2017, 1:13am

After careful investigation I noticed that a lot of sorted queries are running, and I assumed the Harlin's suggestion is correct. There is no way for me to change the queries, so I had to add shards and replicas to the number which pushed down the CPU usage to acceptable level, but still considerably high.