ES Takes CPU Hostage

[mt9999 (Mike T)]

I have a pretty simple ES docker stack deployed with about 4 indexes. One has ~44m objects. I have tried to follow everyones suggestions on here like increase heap space, I even gave it 6 cpu's. It seems no matter what I add, ES will find a way to eat all of the resources. Can someone provide some additional pointers that have worked? Btw the hot threads dump shows:

::: {1CFsBFV}{1CFsBFVXTWqc4bOa8ugrsQ}{fpmCAvUASEu_BMrVjOkXyA}{172.18.0.2}{172.18.0.2:9300}
Hot threads at 2018-08-03T00:31:44.670Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

91.5% (457.6ms out of 500ms) cpu usage by thread 'elasticsearch[1CFsBFV][[index_xxxxxx][3]: Lucene Merge Thread #0]'
5/10 snapshots sharing following 29 elements
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer$4.get(Lucene54DocValuesProducer.java:524)
org.apache.lucene.util.LongValues.get(LongValues.java:45)
org.apache.lucene.index.SingletonSortedNumericDocValues.setDocument(SingletonSortedNumericDocValues.java:52)
org.apache.lucene.codecs.DocValuesConsumer$SortedNumericDocValuesSub.nextDoc(DocValuesConsumer.java:449)
org.apache.lucene.index.DocIDMerger$SequentialDocIDMerger.next(DocIDMerger.java:100)
org.apache.lucene.codecs.DocValuesConsumer$3$1.setNext(DocValuesConsumer.java:511)
org.apache.lucene.codecs.DocValuesConsumer$3$1.hasNext(DocValuesConsumer.java:491)
org.apache.lucene.codecs.DocValuesConsumer$10$1.hasNext(DocValuesConsumer.java:1019)
java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)
java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
java.util.stream.StreamSpliterators$WrappingSpliterator$$Lambda$1542/350734423.getAsBoolean(Unknown Source)

[warkolm (Mark Walkom)]

I can't see any other threads on this problem so it might be good if you can share some extra info so we can help

What version of Elasticsearch?
Which docker image are you using?
How much heap/memory and CPU is assigned?
Are you using Monitoring to see what is happening?
How many nodes, indices, shards and GB of data?

[mt9999 (Mike T)]

Thanks for the quick response! I am taking this over from someone who initially set it up. I'm an ES n00b. I thought a quick +CPU +Memory, assign more heap space would have done it. I will need to find those details and get back to you.

[mt9999 (Mike T)]

@warkolm -
ES stack is all v6.3.2. Single node
System : 6 vCPU, 12GB RAM
4 indexes, 52 Shards, 50GB
No monitoring

[Christian_Dahlqvist]

It seems like merging is the bottleneck. What is the work load against these indices? Are you frequently updating documents or primarily inserting new ones? Are you using parent-child or nested documents? What type of storage do you have (SSD, spinning disk, SAN..)?

[mt9999 (Mike T)]

Its SSD. There are semi frequent updates during the day. I was showing a somewhat idle instance. The documents are not related. There are some nested elements on the docs themselves, but I don't think most of of it needs to be indexed. There are a lot of fields on the documents themselves. I would say ~125