Elasticsearch full CPU utillization

Hi

We have an elasticsearch cluster, which produces a very very high cpu usage
, all the nodes are fully utilize all the time.
When the thread pools (search thread pool specifically) get full, the
cluster stops responding until we hard restart all the nodes.
We'll appreciate any help to avoid thses situations. Details below:

Thanks,
Nitzan.

elasticsearch version - 1.3.4
java version -
Oracle java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

5 nodes
2 Indices
shards - 61
java heap size - 30G
documents number - 485042729

ec2 machines
instance type - c3.8xlarge
vCPU - 32
RAM - 60GB
SSD 1000 iops for data store

Operating system -
Ubuntu precise 12.04

hot threads:

curl -XGET http://localhost:9200/_nodes/hot_threads

::: [ES1][SmFW31WOTH-Nr-0VFpXTag][ES1][inet[
/10.0.1.10:9300]]{max_local_storage_nodes=1}

47.4% (236.9ms out of 500ms) cpu usage by thread
'elasticsearch[ES1][search][T#93]'
6/10 snapshots sharing following 14 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 15 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

42.8% (213.8ms out of 500ms) cpu usage by thread
'elasticsearch[ES1][search][T#85]'
6/10 snapshots sharing following 15 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 15 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

42.7% (213.3ms out of 500ms) cpu usage by thread
'elasticsearch[ES1][search][T#26]'
5/10 snapshots sharing following 16 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:59)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 15 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

::: [ES3][Gb8cjXtWRUShqdnVUhPEqw][ES3][inet[
/10.0.1.11:9300]]{max_local_storage_nodes=1}

42.2% (211.1ms out of 500ms) cpu usage by thread
'elasticsearch[ES3][search][T#15]'
7/10 snapshots sharing following 16 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 15 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

42.2% (210.8ms out of 500ms) cpu usage by thread
'elasticsearch[ES3][search][T#67]'
3/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:59)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:170)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 18 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:170)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:170)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

41.9% (209.6ms out of 500ms) cpu usage by thread
'elasticsearch[ES3][search][T#66]'
8/10 snapshots sharing following 18 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

::: [ES5][q_VWePETRIOLS6IxvK3CxQ][ES5][inet[
/10.0.1.12:9300]]{max_local_storage_nodes=1}

38.3% (191.4ms out of 500ms) cpu usage by thread
'elasticsearch[ES5][search][T#40]'
7/10 snapshots sharing following 15 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 3 elements
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)

37.5% (187.2ms out of 500ms) cpu usage by thread
'elasticsearch[ES5][search][T#15]'
2/10 snapshots sharing following 18 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:
163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.se

...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Add more nodes :slight_smile:

On 4 March 2015 at 01:29, Nitzan Dana nitzan@adallom.com wrote:

Hi

We have an elasticsearch cluster, which produces a very very high cpu
usage , all the nodes are fully utilize all the time.
When the thread pools (search thread pool specifically) get full, the
cluster stops responding until we hard restart all the nodes.
We'll appreciate any help to avoid thses situations. Details below:

Thanks,
Nitzan.

elasticsearch version - 1.3.4
java version -
Oracle java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

5 nodes
2 Indices
shards - 61
java heap size - 30G
documents number - 485042729

ec2 machines
instance type - c3.8xlarge
vCPU - 32
RAM - 60GB
SSD 1000 iops for data store

Operating system -
Ubuntu precise 12.04

hot threads:

curl -XGET http://localhost:9200/_nodes/hot_threads

::: [ES1][SmFW31WOTH-Nr-0VFpXTag][ES1][inet[/10.0.1.10:9300
]]{max_local_storage_nodes=1}

47.4% (236.9ms out of 500ms) cpu usage by thread
'elasticsearch[ES1][search][T#93]'
6/10 snapshots sharing following 14 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run
(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 15 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run
(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

42.8% (213.8ms out of 500ms) cpu usage by thread
'elasticsearch[ES1][search][T#85]'
6/10 snapshots sharing following 15 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run
(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 15 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run
(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

42.7% (213.3ms out of 500ms) cpu usage by thread
'elasticsearch[ES1][search][T#26]'
5/10 snapshots sharing following 16 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:59)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run
(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 15 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:206)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call
(SearchServiceTransportAction.java:203)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run
(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

::: [ES3][Gb8cjXtWRUShqdnVUhPEqw][ES3][inet[/10.0.1.11:9300
]]{max_local_storage_nodes=1}

42.2% (211.1ms out of 500ms) cpu usage by thread
'elasticsearch[ES3][search][T#15]'
7/10 snapshots sharing following 16 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 15 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

42.2% (210.8ms out of 500ms) cpu usage by thread
'elasticsearch[ES3][search][T#67]'
3/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:59)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:170)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
149)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 18 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:170)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
149)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:170)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
149)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

41.9% (209.6ms out of 500ms) cpu usage by thread
'elasticsearch[ES3][search][T#66]'
8/10 snapshots sharing following 18 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
149)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
149)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

::: [ES5][q_VWePETRIOLS6IxvK3CxQ][ES5][inet[/10.0.1.12:9300
]]{max_local_storage_nodes=1}

38.3% (191.4ms out of 500ms) cpu usage by thread
'elasticsearch[ES5][search][T#40]'
7/10 snapshots sharing following 15 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
309)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
116)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 3 elements
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

37.5% (187.2ms out of 500ms) cpu usage by thread
'elasticsearch[ES5][search][T#15]'
2/10 snapshots sharing following 18 elements
org.elasticsearch.common.lucene.search.FilteredCollector.collect(
FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:
149)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:262)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:688)
org.elasticsearch.search.action.
SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(
SearchServiceTransportAction.java:677)
org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 19 elements
org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
FilteredDocIdSetIterator.java:60)
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(
ConstantScoreQuery.java:257)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.
java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java
:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(
ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
533)
org.apache.lucene.se

...

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f7f9112-2011-49ba-b382-1886c8216602%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8bmRow9tekMwKD7_8iSyxnriO-QBeZibyKx79KA00WYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.