# Elasticsearch full CPU utillization **URL:** \ **Category:** Elasticsearch\ **Created:** [March 3, 2015, 2:29pm UTC](https://discuss.elastic.co/t/elasticsearch-full-cpu-utillization/22490 "2015-03-03T14:29:26Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Nitzan\_Dana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitzan_dana/32/11140_2.png) [@Nitzan\_Dana](https://discuss.elastic.co/u/Nitzan_Dana)\ **Post date:** [March 3, 2015, 2:29pm UTC](https://discuss.elastic.co/t/elasticsearch-full-cpu-utillization/22490/1 "2015-03-03T14:29:26Z")
Hi We have an elasticsearch cluster, which produces a very very high cpu usage , all the nodes are fully utilize all the time. When the thread pools (search thread pool specifically) get full, the cluster stops responding until we hard restart all the nodes. We'll appreciate any help to avoid thses situations. Details below: Thanks, Nitzan. elasticsearch version - 1.3.4 java version - Oracle java version "1.7.0\_67" Java(TM) SE Runtime Environment (build 1.7.0\_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) 5 nodes 2 Indices shards - 61 java heap size - 30G documents number - 485042729 ec2 machines instance type - c3.8xlarge vCPU - 32 RAM - 60GB SSD 1000 iops for data store Operating system - Ubuntu precise 12.04 hot threads: curl -XGET [http://localhost:9200/\_nodes/hot\_threads](http://localhost:9200/_nodes/hot_threads) ::: [ES1][SmFW31WOTH-Nr-0VFpXTag][ES1][inet[ /10.0.1.10:9300]]{max\_local\_storage\_nodes=1} 47.4% (236.9ms out of 500ms) cpu usage by thread 'elasticsearch[ES1][search][T#93]' 6/10 snapshots sharing following 14 elements org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:206) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:203) org.elasticsearch.search.action.SearchServiceTransportAction$23.run( SearchServiceTransportAction.java:517) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 4/10 snapshots sharing following 15 elements org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:206) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:203) org.elasticsearch.search.action.SearchServiceTransportAction$23.run( SearchServiceTransportAction.java:517) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 42.8% (213.8ms out of 500ms) cpu usage by thread 'elasticsearch[ES1][search][T#85]' 6/10 snapshots sharing following 15 elements org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:206) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:203) org.elasticsearch.search.action.SearchServiceTransportAction$23.run( SearchServiceTransportAction.java:517) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 4/10 snapshots sharing following 15 elements org.elasticsearch.common.lucene.search.FilteredCollector.collect( FilteredCollector.java:60) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:206) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:203) org.elasticsearch.search.action.SearchServiceTransportAction$23.run( SearchServiceTransportAction.java:517) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 42.7% (213.3ms out of 500ms) cpu usage by thread 'elasticsearch[ES1][search][T#26]' 5/10 snapshots sharing following 16 elements org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( FilteredDocIdSetIterator.java:59) org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:206) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:203) org.elasticsearch.search.action.SearchServiceTransportAction$23.run( SearchServiceTransportAction.java:517) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 5/10 snapshots sharing following 15 elements org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:206) org.elasticsearch.search.action.SearchServiceTransportAction$5.call( SearchServiceTransportAction.java:203) org.elasticsearch.search.action.SearchServiceTransportAction$23.run( SearchServiceTransportAction.java:517) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) ::: [ES3][Gb8cjXtWRUShqdnVUhPEqw][ES3][inet[ /10.0.1.11:9300]]{max\_local\_storage\_nodes=1} 42.2% (211.1ms out of 500ms) cpu usage by thread 'elasticsearch[ES3][search][T#15]' 7/10 snapshots sharing following 16 elements org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( FilteredDocIdSetIterator.java:60) org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 3/10 snapshots sharing following 15 elements org.elasticsearch.common.lucene.search.FilteredCollector.collect( FilteredCollector.java:60) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 42.2% (210.8ms out of 500ms) cpu usage by thread 'elasticsearch[ES3][search][T#67]' 3/10 snapshots sharing following 19 elements org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( FilteredDocIdSetIterator.java:59) org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:170) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 5/10 snapshots sharing following 18 elements org.elasticsearch.common.lucene.search.FilteredCollector.collect( FilteredCollector.java:60) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:170) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 2/10 snapshots sharing following 19 elements org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( FilteredDocIdSetIterator.java:60) org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:170) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 41.9% (209.6ms out of 500ms) cpu usage by thread 'elasticsearch[ES3][search][T#66]' 8/10 snapshots sharing following 18 elements org.elasticsearch.common.lucene.search.FilteredCollector.collect( FilteredCollector.java:60) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 2/10 snapshots sharing following 19 elements org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( FilteredDocIdSetIterator.java:60) org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) ::: [ES5][q\_VWePETRIOLS6IxvK3CxQ][ES5][inet[ /10.0.1.12:9300]]{max\_local\_storage\_nodes=1} 38.3% (191.4ms out of 500ms) cpu usage by thread 'elasticsearch[ES5][search][T#40]' 7/10 snapshots sharing following 15 elements org.elasticsearch.common.lucene.search.FilteredCollector.collect( FilteredCollector.java:60) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 3/10 snapshots sharing following 3 elements java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 37.5% (187.2ms out of 500ms) cpu usage by thread 'elasticsearch[ES5][search][T#15]' 2/10 snapshots sharing following 18 elements org.elasticsearch.common.lucene.search.FilteredCollector.collect( FilteredCollector.java:60) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:193) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149 ) org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:262) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:688) org.elasticsearch.search.action. SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( SearchServiceTransportAction.java:677) org.elasticsearch.transport.netty. MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) java.lang.Thread.run(Thread.java:745) 5/10 snapshots sharing following 19 elements org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( FilteredDocIdSetIterator.java:60) org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( ConstantScoreQuery.java:257) org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. java:192) org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java: 163) org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) org.elasticsearch.search.internal.ContextIndexSearcher.search( ContextIndexSearcher.java:175) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533) org.apache.lucene.se ... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2015, 9:29pm UTC](https://discuss.elastic.co/t/elasticsearch-full-cpu-utillization/22490/2 "2015-03-03T21:29:40Z")
Add more nodes 🙂 On 4 March 2015 at 01:29, Nitzan Dana [nitzan@adallom.com](mailto:nitzan@adallom.com) wrote: > Hi > > We have an elasticsearch cluster, which produces a very very high cpu > usage , all the nodes are fully utilize all the time. > When the thread pools (search thread pool specifically) get full, the > cluster stops responding until we hard restart all the nodes. > We'll appreciate any help to avoid thses situations. Details below: > > Thanks, > Nitzan. > > elasticsearch version - 1.3.4 > java version - > Oracle java version "1.7.0\_67" > Java(TM) SE Runtime Environment (build 1.7.0\_67-b01) > Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) > > 5 nodes > 2 Indices > shards - 61 > java heap size - 30G > documents number - 485042729 > > ec2 machines > instance type - c3.8xlarge > vCPU - 32 > RAM - 60GB > SSD 1000 iops for data store > > Operating system - > Ubuntu precise 12.04 > > hot threads: > > curl -XGET [http://localhost:9200/\_nodes/hot\_threads](http://localhost:9200/_nodes/hot_threads) > > ::: [ES1][SmFW31WOTH-Nr-0VFpXTag][ES1][inet[/10.0.1.10:9300 > ]]{max\_local\_storage\_nodes=1} > > 47.4% (236.9ms out of 500ms) cpu usage by thread > 'elasticsearch[ES1][search][T#93]' > 6/10 snapshots sharing following 14 elements > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:206) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:203) > org.elasticsearch.search.action.SearchServiceTransportAction$23.run > (SearchServiceTransportAction.java:517) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 4/10 snapshots sharing following 15 elements > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:206) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:203) > org.elasticsearch.search.action.SearchServiceTransportAction$23.run > (SearchServiceTransportAction.java:517) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 42.8% (213.8ms out of 500ms) cpu usage by thread > 'elasticsearch[ES1][search][T#85]' > 6/10 snapshots sharing following 15 elements > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:206) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:203) > org.elasticsearch.search.action.SearchServiceTransportAction$23.run > (SearchServiceTransportAction.java:517) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 4/10 snapshots sharing following 15 elements > org.elasticsearch.common.lucene.search.FilteredCollector.collect( > FilteredCollector.java:60) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:206) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:203) > org.elasticsearch.search.action.SearchServiceTransportAction$23.run > (SearchServiceTransportAction.java:517) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 42.7% (213.3ms out of 500ms) cpu usage by thread > 'elasticsearch[ES1][search][T#26]' > 5/10 snapshots sharing following 16 elements > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( > FilteredDocIdSetIterator.java:59) > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:206) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:203) > org.elasticsearch.search.action.SearchServiceTransportAction$23.run > (SearchServiceTransportAction.java:517) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 5/10 snapshots sharing following 15 elements > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:206) > org.elasticsearch.search.action.SearchServiceTransportAction$5.call > (SearchServiceTransportAction.java:203) > org.elasticsearch.search.action.SearchServiceTransportAction$23.run > (SearchServiceTransportAction.java:517) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > ::: [ES3][Gb8cjXtWRUShqdnVUhPEqw][ES3][inet[/10.0.1.11:9300 > ]]{max\_local\_storage\_nodes=1} > > 42.2% (211.1ms out of 500ms) cpu usage by thread > 'elasticsearch[ES3][search][T#15]' > 7/10 snapshots sharing following 16 elements > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( > FilteredDocIdSetIterator.java:60) > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 3/10 snapshots sharing following 15 elements > org.elasticsearch.common.lucene.search.FilteredCollector.collect( > FilteredCollector.java:60) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 42.2% (210.8ms out of 500ms) cpu usage by thread > 'elasticsearch[ES3][search][T#67]' > 3/10 snapshots sharing following 19 elements > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( > FilteredDocIdSetIterator.java:59) > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:170) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 510) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 345) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 149) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 5/10 snapshots sharing following 18 elements > org.elasticsearch.common.lucene.search.FilteredCollector.collect( > FilteredCollector.java:60) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:170) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 510) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 345) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 149) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 2/10 snapshots sharing following 19 elements > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( > FilteredDocIdSetIterator.java:60) > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:170) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 510) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 345) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 149) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 41.9% (209.6ms out of 500ms) cpu usage by thread > 'elasticsearch[ES3][search][T#66]' > 8/10 snapshots sharing following 18 elements > org.elasticsearch.common.lucene.search.FilteredCollector.collect( > FilteredCollector.java:60) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 510) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 345) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 149) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 2/10 snapshots sharing following 19 elements > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( > FilteredDocIdSetIterator.java:60) > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 510) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 345) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 149) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > ::: [ES5][q\_VWePETRIOLS6IxvK3CxQ][ES5][inet[/10.0.1.12:9300 > ]]{max\_local\_storage\_nodes=1} > > 38.3% (191.4ms out of 500ms) cpu usage by thread > 'elasticsearch[ES5][search][T#40]' > 7/10 snapshots sharing following 15 elements > org.elasticsearch.common.lucene.search.FilteredCollector.collect( > FilteredCollector.java:60) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 309) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 116) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 3/10 snapshots sharing following 3 elements > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 37.5% (187.2ms out of 500ms) cpu usage by thread > 'elasticsearch[ES5][search][T#15]' > 2/10 snapshots sharing following 18 elements > org.elasticsearch.common.lucene.search.FilteredCollector.collect( > FilteredCollector.java:60) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:193) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 510) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 345) > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java: > 149) > org.elasticsearch.search.SearchService.executeQueryPhase( > SearchService.java:262) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:688) > org.elasticsearch.search.action. > SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived( > SearchServiceTransportAction.java:677) > org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 5/10 snapshots sharing following 19 elements > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( > FilteredDocIdSetIterator.java:60) > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc( > ConstantScoreQuery.java:257) > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight. > java:192) > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java > :163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 621) > org.elasticsearch.search.internal.ContextIndexSearcher.search( > ContextIndexSearcher.java:175) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 581) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java: > 533) > org.apache.lucene.se > > ... > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8bmRow9tekMwKD7\_8iSyxnriO-QBeZibyKx79KA00WYA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8bmRow9tekMwKD7_8iSyxnriO-QBeZibyKx79KA00WYA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:28am UTC](https://discuss.elastic.co/t/elasticsearch-full-cpu-utillization/22490/3 "2017-07-06T00:28:48Z")