100 percent cpu utilization on single node elasticsearch

Elastic Stack Elasticsearch
1

Hi Team,

Facing 100 cpu utilization when running multiple search requests with only 10 users on an AWS Windows server, with 2 core cpu and 16 GB RAM, with 2 GB allocated as HEAP in elasticsearch. 

Can anyone suggest how to lower the cpu utilization and improve search performance??
Should number of cores be increased because it is not consuming much heap??

Output of hot threads API is below:

::: {19795APPUAT}{f9Pg7xA5SuSKvkMueAsvEQ}{49PUBdiEQ7iiHlWjOHg_YQ}{10.0.2.196}{10.0.2.196:9300}{xpack.installed=true}
   Hot threads at 2020-04-02T08:00:46.105Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   137.5% (687.5ms out of 500ms) cpu usage by thread 'elasticsearch[19795APPUAT][search][T#4]'
     2/10 snapshots sharing following 27 elements
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:66)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:310)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:228)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:246)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:246)
       org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:667)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.extractTerms(UnifiedHighlighter.java:147)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:569)
       org.apache.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:107)
       org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:118)
       org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase.hitExecute(HighlightPhase.java:107)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:169)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:389)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
       org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       org.elasticsearch.search.SearchService$$Lambda$3432/629038329.apply(Unknown Source)
       org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145)
       org.elasticsearch.action.ActionListener$$Lambda$3433/988640122.accept(Unknown Source)
       org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
       org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 26 elements
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:310)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:228)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:246)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:246)
       org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:667)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.extractTerms(UnifiedHighlighter.java:147)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:569)
       org.apache.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:107)
       org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:118)
       org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase.hitExecute(HighlightPhase.java:107)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:169)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:389)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
       org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       org.elasticsearch.search.SearchService$$Lambda$3432/629038329.apply(Unknown Source)
       org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145)
       org.elasticsearch.action.ActionListener$$Lambda$3433/988640122.accept(Unknown Source)
       org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
       org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     4/10 snapshots sharing following 29 elements
       org.apache.lucene.search.BooleanQuery.<init>(BooleanQuery.java:42)
       org.apache.lucene.search.BooleanQuery$Builder.build(BooleanQuery.java:133)
       org.apache.lucene.search.BlendedTermQuery$1.rewrite(BlendedTermQuery.java:140)
       org.apache.lucene.search.BlendedTermQuery.rewrite(BlendedTermQuery.java:294)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:228)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:246)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:246)
       org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:668)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.extractTerms(UnifiedHighlighter.java:147)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:569)
       org.apache.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:107)
       org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:118)
       org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase.hitExecute(HighlightPhase.java:107)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:169)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:389)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
       org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       org.elasticsearch.search.SearchService$$Lambda$3432/629038329.apply(Unknown Source)
       org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145)
       org.elasticsearch.action.ActionListener$$Lambda$3433/988640122.accept(Unknown Source)
       org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
       org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 19 elements
       org.apache.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:107)
       org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:118)
       org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase.hitExecute(HighlightPhase.java:107)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:169)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:389)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
       org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       org.elasticsearch.search.SearchService$$Lambda$3432/629038329.apply(Unknown Source)
       org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145)
       org.elasticsearch.action.ActionListener$$Lambda$3433/988640122.accept(Unknown Source)
       org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
       org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
   
   118.8% (593.7ms out of 500ms) cpu usage by thread 'elasticsearch[19795APPUAT][search][T#2]'
     2/10 snapshots sharing following 29 elements
       org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:74)
       org.apache.lucene.util.automaton.CharacterRunAutomaton.<init>(CharacterRunAutomaton.java:39)
       org.apache.lucene.util.automaton.CharacterRunAutomaton.<init>(CharacterRunAutomaton.java:28)
       org.apache.lucene.search.uhighlight.MultiTermHighlighting$1.<init>(MultiTermHighlighting.java:127)
       org.apache.lucene.search.uhighlight.MultiTermHighlighting.extractAutomata(MultiTermHighlighting.java:127)
       org.apache.lucene.search.uhighlight.MultiTermHighlighting.extractAutomata(MultiTermHighlighting.java:91)
       org.apache.lucene.search.uhighlight.MultiTermHighlighting.extractAutomata(MultiTermHighlighting.java:77)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.getAutomata(UnifiedHighlighter.java:819)
       org.apache.lucene.search.uhighlight.CustomUnifiedHighlighter.getFieldHighlighter(CustomUnifiedHighlighter.java:144)
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:574)
       org.apache.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:107)
       org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:118)
       org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase.hitExecute(HighlightPhase.java:107)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:169)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:389)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
       org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       org.elasticsearch.search.SearchService$$Lambda$3432/629038329.apply(Unknown Source)
     
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     7/10 snapshots sharing following 21 elements
       org.apache.lucene.search.uhighlight.UnifiedHighlighter.extractTerms(UnifiedHighlighter.j
2

The hot_threads output seems to indicate that highlighting is causing your CPU load. You could try to disable highlighting first and see if your CPU load goes down.

Then the next step would be to figure out how to adapt to this problem (adding CPUs, adding nodes, rethinking the highlighting query if you may not need to do that on big fields, etc).

3 
I have used term vectors with "with_positions_offsets" in document content field and using unified highlighter but experiencing too much slowness. 

Please suggest solutions for this.
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.