ES Takes CPU Hostage

Elastic Stack Elasticsearch
1

I have a pretty simple ES docker stack deployed with about 4 indexes. One has ~44m objects. I have tried to follow everyones suggestions on here like increase heap space, I even gave it 6 cpu's. It seems no matter what I add, ES will find a way to eat all of the resources. Can someone provide some additional pointers that have worked? Btw the hot threads dump shows:

::: {1CFsBFV}{1CFsBFVXTWqc4bOa8ugrsQ}{fpmCAvUASEu_BMrVjOkXyA}{172.18.0.2}{172.18.0.2:9300}
Hot threads at 2018-08-03T00:31:44.670Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

91.5% (457.6ms out of 500ms) cpu usage by thread 'elasticsearch[1CFsBFV][[index_xxxxxx][3]: Lucene Merge Thread #0]'
5/10 snapshots sharing following 29 elements
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer$4.get(Lucene54DocValuesProducer.java:524)
org.apache.lucene.util.LongValues.get(LongValues.java:45)
org.apache.lucene.index.SingletonSortedNumericDocValues.setDocument(SingletonSortedNumericDocValues.java:52)
org.apache.lucene.codecs.DocValuesConsumer$SortedNumericDocValuesSub.nextDoc(DocValuesConsumer.java:449)
org.apache.lucene.index.DocIDMerger$SequentialDocIDMerger.next(DocIDMerger.java:100)
org.apache.lucene.codecs.DocValuesConsumer$3$1.setNext(DocValuesConsumer.java:511)
org.apache.lucene.codecs.DocValuesConsumer$3$1.hasNext(DocValuesConsumer.java:491)
org.apache.lucene.codecs.DocValuesConsumer$10$1.hasNext(DocValuesConsumer.java:1019)
java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)
java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
java.util.stream.StreamSpliterators$WrappingSpliterator$$Lambda$1542/350734423.getAsBoolean(Unknown Source)

2

I can't see any other threads on this problem so it might be good if you can share some extra info so we can help :slight_smile:

What version of Elasticsearch?
Which docker image are you using?
How much heap/memory and CPU is assigned?
Are you using Monitoring to see what is happening?
How many nodes, indices, shards and GB of data?

3

Thanks for the quick response! I am taking this over from someone who initially set it up. I'm an ES n00b. I thought a quick +CPU +Memory, assign more heap space would have done it. I will need to find those details and get back to you.

1 Like
4

@warkolm -
ES stack is all v6.3.2. Single node
System : 6 vCPU, 12GB RAM
4 indexes, 52 Shards, 50GB
No monitoring

5

It seems like merging is the bottleneck. What is the work load against these indices? Are you frequently updating documents or primarily inserting new ones? Are you using parent-child or nested documents? What type of storage do you have (SSD, spinning disk, SAN..)?

6

Its SSD. There are semi frequent updates during the day. I was showing a somewhat idle instance. The documents are not related. There are some nested elements on the docs themselves, but I don't think most of of it needs to be indexed. There are a lot of fields on the documents themselves. I would say ~125

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.