# CPU 100% utilization

**URL:** <https://discuss.elastic.co/t/cpu-100-utilization/23080>\
**Category:** Elasticsearch\
**Created:** [April 3, 2015, 10:55am UTC](https://discuss.elastic.co/t/cpu-100-utilization/23080 "2015-04-03T10:55:20Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyrilforce](https://avatars.discourse-cdn.com/v4/letter/c/76d3ee/32.png) [@cyrilforce](https://discuss.elastic.co/u/cyrilforce)\
**Post date:** [April 3, 2015, 10:55am UTC](https://discuss.elastic.co/t/cpu-100-utilization/23080/1 "2015-04-03T10:55:20Z")

</div>

Hi,

Having to test the performance of the ES with some load testing it reaches  
100% CPU utilization and following is hot threads :

## ES configuration

ES\_MIN\_MEM 256m  
ES\_MAX\_MEM 1g

## Load testing configuration

From 100 threads per second ramping up to 400 threads per second over  
period of 60 second and it starts failing we reaches 300+ threads per  
seconds

## Hot threads

:/elasticsearch/logs ]$ curl -XGET  
'[http://localhost:9200//\_nodes/hot\_threads?pretty](http://localhost:9200//_nodes/hot_threads?pretty)'  
:: [Mesmero][if9uerQWRVGmpD3iq0ELYg][2905f3223ee8][inet[/172.17.0.62:9300]]

37.6% (188ms out of 500ms) cpu usage by thread  
'elasticsearch[Mesmero][search][T#1]'  
2/10 snapshots sharing following 6 elements  
java.lang.System.currentTimeMillis(Native Method)

org.elasticsearch.action.search.type.AbstractAsyncAction.buildTookInMillis(AbstractAsyncAction.java:46)

org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
4/10 snapshots sharing following 6 elements

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
3/10 snapshots sharing following 18 elements

org.apache.lucene.util.AttributeSource.clearAttributes(AttributeSource.java:279)

org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:194)

org.apache.lucene.analysis.standard.StandardFilter.incrementToken(StandardFilter.java:57)

org.apache.lucene.analysis.en.EnglishPossessiveFilter.incrementToken(EnglishPossessiveFilter.java:57)

org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62)

org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:90)

org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:91)

org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218)

org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)

org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)  
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
unique snapshot  
sun.nio.ch.NativeThread.current(Native Method)  
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)  
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)  
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)

org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)

org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)

org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)

org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)  
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101)

org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)

org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:354)  
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)

org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110)  
org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196)

org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228)  
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)

34.0% (169.9ms out of 500ms) cpu usage by thread  
'elasticsearch[Mesmero][search][T#3]'  
2/10 snapshots sharing following 13 elements  
java.util.IdentityHashMap.entrySet(IdentityHashMap.java:1098)

org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:251)

org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:230)

org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:138)

org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:262)

org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:91)

org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:68)

org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)

org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)

org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
3/10 snapshots sharing following 20 elements

org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190)

org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)

org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)

org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)

org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)  
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
3/10 snapshots sharing following 12 elements

org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:96)

org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218)

org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)

org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)  
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
2/10 snapshots sharing following 10 elements

org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)

org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)  
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)

22.8% (114.1ms out of 500ms) cpu usage by thread  
'elasticsearch[Mesmero][search][T#4]'  
5/10 snapshots sharing following 36 elements  
java.lang.Throwable.fillInStackTrace(Native Method)  
java.lang.Throwable.fillInStackTrace(Throwable.java:783)  
java.lang.Throwable.(Throwable.java:250)  
java.lang.Exception.(Exception.java:54)  
java.lang.RuntimeException.(RuntimeException.java:51)

org.antlr.v4.runtime.RecognitionException.(RecognitionException.java:65)

org.antlr.v4.runtime.NoViableAltException.(NoViableAltException.java:70)

org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:1776)

org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:496)

org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:423)

com.test.SecurityExpressionParser.security\_rule(SecurityExpressionParser.java:284)

com.test.SecurityExpressionParser.security\_rule(SecurityExpressionParser.java:310)

com.test.SecurityExpressionParser.security\_rule(SecurityExpressionParser.java:227)  
com.test.ExpressionManager.compile(ExpressionManager.java:134)

com.test.ExpressionManager.valiateExpression(ExpressionManager.java:293)  
com.test.VisibilityFilter.run(VisibilityFilter.java:107)

org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190)

org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)

org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)

org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)

org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)  
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
4/10 snapshots sharing following 7 elements

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)  
unique snapshot  
java.lang.System.nanoTime(Native Method)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:487)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

Anyone have any idea ?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5d52ccbd-8777-47c9-855c-6a96d9e0a944%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d52ccbd-8777-47c9-855c-6a96d9e0a944%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![aaronmefford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaronmefford/32/460_2.png) [@aaronmefford](https://discuss.elastic.co/u/aaronmefford)\
**Post date:** [April 3, 2015, 4:08pm UTC](https://discuss.elastic.co/t/cpu-100-utilization/23080/2 "2015-04-03T16:08:18Z")

</div>

Thats a lot of threads for 1G of memory.

On Friday, April 3, 2015 at 4:55:20 AM UTC-6, cyrilforce wrote:

> Hi,
> 
> Having to test the performance of the ES with some load testing it  
> reaches 100% CPU utilization and following is hot threads :
> 
> ## ES configuration
> 
> ES\_MIN\_MEM 256m  
> ES\_MAX\_MEM 1g
> 
> ## Load testing configuration
> 
> From 100 threads per second ramping up to 400 threads per second over  
> period of 60 second and it starts failing we reaches 300+ threads per  
> seconds
> 
> ## Hot threads
> 
> :/elasticsearch/logs ]$ curl -XGET '  
> [http://localhost:9200//\_nodes/hot\_threads?pretty](http://localhost:9200//_nodes/hot_threads?pretty)'  
> :: [Mesmero][if9uerQWRVGmpD3iq0ELYg][2905f3223ee8][inet[/172.17.0.62:9300  
> ]]
> 
> 37.6% (188ms out of 500ms) cpu usage by thread  
> 'elasticsearch[Mesmero][search][T#1]'  
> 2/10 snapshots sharing following 6 elements  
> java.lang.System.currentTimeMillis(Native Method)
> 
> org.elasticsearch.action.search.type.AbstractAsyncAction.buildTookInMillis(AbstractAsyncAction.java:46)
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 4/10 snapshots sharing following 6 elements
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 3/10 snapshots sharing following 18 elements
> 
> org.apache.lucene.util.AttributeSource.clearAttributes(AttributeSource.java:279)
> 
> org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:194)
> 
> org.apache.lucene.analysis.standard.StandardFilter.incrementToken(StandardFilter.java:57)
> 
> org.apache.lucene.analysis.en.EnglishPossessiveFilter.incrementToken(EnglishPossessiveFilter.java:57)
> 
> org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62)
> 
> org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:90)
> 
> org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:91)
> 
> org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218)
> 
> org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
> 
> org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
> 
> org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> unique snapshot  
> sun.nio.ch.NativeThread.current(Native Method)  
> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)  
> sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)  
> sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
> 
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
> 
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
> 
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
> 
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)  
> org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101)
> 
> org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)
> 
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:354)
> 
> org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)
> 
> org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110)  
> org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196)
> 
> org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228)
> 
> org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)
> 
> 34.0% (169.9ms out of 500ms) cpu usage by thread  
> 'elasticsearch[Mesmero][search][T#3]'  
> 2/10 snapshots sharing following 13 elements  
> java.util.IdentityHashMap.entrySet(IdentityHashMap.java:1098)
> 
> org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:251)
> 
> org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:230)
> 
> org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:138)
> 
> org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:262)
> 
> org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:91)
> 
> org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:68)
> 
> org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
> 
> org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 3/10 snapshots sharing following 20 elements
> 
> org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190)
> 
> org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)
> 
> org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)
> 
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
> 
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> 
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> 
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
> 
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 3/10 snapshots sharing following 12 elements
> 
> org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:96)
> 
> org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218)
> 
> org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
> 
> org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
> 
> org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 2/10 snapshots sharing following 10 elements
> 
> org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
> 
> org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
> 
> org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)
> 
> 22.8% (114.1ms out of 500ms) cpu usage by thread  
> 'elasticsearch[Mesmero][search][T#4]'  
> 5/10 snapshots sharing following 36 elements  
> java.lang.Throwable.fillInStackTrace(Native Method)  
> java.lang.Throwable.fillInStackTrace(Throwable.java:783)  
> java.lang.Throwable.(Throwable.java:250)  
> java.lang.Exception.(Exception.java:54)  
> java.lang.RuntimeException.(RuntimeException.java:51)
> 
> org.antlr.v4.runtime.RecognitionException.(RecognitionException.java:65)
> 
> org.antlr.v4.runtime.NoViableAltException.(NoViableAltException.java:70)
> 
> org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:1776)
> 
> org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:496)
> 
> org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:423)
> 
> com.test.SecurityExpressionParser.security\_rule(SecurityExpressionParser.java:284)
> 
> com.test.SecurityExpressionParser.security\_rule(SecurityExpressionParser.java:310)
> 
> com.test.SecurityExpressionParser.security\_rule(SecurityExpressionParser.java:227)  
> com.test.ExpressionManager.compile(ExpressionManager.java:134)
> 
> com.test.ExpressionManager.valiateExpression(ExpressionManager.java:293)  
> com.test.VisibilityFilter.run(VisibilityFilter.java:107)
> 
> org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190)
> 
> org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)
> 
> org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)
> 
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
> 
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> 
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> 
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
> 
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 4/10 snapshots sharing following 7 elements
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> unique snapshot  
> java.lang.System.nanoTime(Native Method)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:487)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 
> Anyone have any idea ?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c9abadde-e489-467c-ada7-062bd7463ea2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c9abadde-e489-467c-ada7-062bd7463ea2%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:21am UTC](https://discuss.elastic.co/t/cpu-100-utilization/23080/3 "2017-07-06T00:21:48Z")

</div>