ES system CPU spikes to 100% despite of having small amount of data

Elastic Stack Elasticsearch
1

Hello,

I am using Elastic Search v5.2.2.
I have a single node cluster with 3 primary shards & 0 replica shard.
Total documents are around 10 million with 5 GB of data.

I have disabled scoring and most of the fields in my index are not_analyzed.
I am also using the pre-load feature where I load all the files (index.store.preload : ["*"]) before beginning the search operations.

Currently, I have 128 GB RAM and 1.5 TB of hard disk....Heap size is set to 8 GB.

Inspite of having such a small amount of data and a high performance machine, my CPU spikes to 100% and performance is not as expected.

What might be the reason for such slow performance?
And what can be interpreted from the following hot thread output? I can see Scorers in the output even though I have disabled scoring.

Thanks :slight_smile:

2 
::: {sys217}{NDgCEjjcT9CT32i4JlMmWQ}{FILT-0ulRc64Osy1EIUICw}{172.24.22.217}{172.24.22.217:9300}
   Hot threads at 2017-03-19T13:30:33.282Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   81.1% (405.6ms out of 500ms) cpu usage by thread 'elasticsearch[sys217][search][T#12]'
     2/10 snapshots sharing following 23 elements
       org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:500)
       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:176)
       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:208)
       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)
       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)
       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:246)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:260)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:298)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 31 elements
       org.apache.lucene.util.automaton.Operations.determinize(Operations.java:728)
       org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1163)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:238)
       org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:102)
       org.apache.lucene.search.TermRangeQuery.<init>(TermRangeQuery.java:66)
       org.elasticsearch.index.mapper.StringFieldType.rangeQuery(StringFieldType.java:93)
       org.elasticsearch.index.query.RangeQueryBuilder.doToQuery(RangeQueryBuilder.java:517)
       org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 19 elements
       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)
       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)
       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:260)
                  org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:295)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 26 elements
       java.lang.Throwable.fillInStackTrace(Native Method)
       java.lang.Throwable.fillInStackTrace(Throwable.java:783)
       java.lang.Throwable.<init>(Throwable.java:250)
       java.lang.Error.<init>(Error.java:58)

       org.apache.lucene.queryparser.classic.MapperQueryParser.<init>(MapperQueryParser.java:86)
       org.elasticsearch.index.query.QueryShardContext.<init>(QueryShardContext.java:94)
       org.elasticsearch.index.IndexService.newQueryShardContext(IndexService.java:470)
       org.elasticsearch.search.DefaultSearchContext.<init>(DefaultSearchContext.java:171)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:298)
       
       org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1163)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:238)
       org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:102)
       org.apache.lucene.search.TermRangeQuery.<init>(TermRangeQuery.java:66)
       org.elasticsearch.index.mapper.StringFieldType.rangeQuery(StringFieldType.java:93)
       org.elasticsearch.index.query.RangeQueryBuilder.doToQuery(RangeQueryBuilder.java:517)
       org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
3

What sort of searches are you doing?

4

I am using multisearch where i have 5 individual queries on different fields.
All the queries perform search on a single index.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.