.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f0e8a4ad-9d5a-42d8-b69e-fb4ccaf75c26%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0e8a4ad-9d5a-42d8-b69e-fb4ccaf75c26%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Graham\_Lenton](https://discuss.elastic.co/u/Graham_Lenton)
#### Post date: [March 3, 2015, 7:01am UTC](https://discuss.elastic.co/t/high-cpu-during-scroll-large-request-low-iowait/22435/6 "2015-03-03T07:01:19Z")
Justin - also keep an eye on your client nodes - they will be doing quite a
bit of garbage collection, which will only increase as paging gets deeper.
Bear in mind that your client node has to reduce (paging\_depth \*
number\_of\_shards \* number\_of\_indices) down to number\_of\_hits required for
each and every request.
So retrieving results 400,000 to 407,500 from a single index with 15 shards
will give the client node 400,000 \* 15 \* 1 == 6,000,000 results to crunch
down to your 7,500. That will result in a great deal of GC.
If this is the case, possible strategies to improve matters are:
- keep the client heap size as small as possible to make GC happen more
frequently - this depends on the type of requests and concurrency you have.
- reindex with less shards if possible - experiment with a single shard
with your data on a single node of your hardware to find how many docs
_you_ can fit in a shard before performance hits the wall.
- ensure your application uses all 3 client nodes to spread the load, and
increase the number of client nodes if possible
- have more than one index - if you can segment the query in a sensible
way e.g. time series data
G
On Tuesday, 3 March 2015 02:21:37 UTC, Justin Zhu wrote:
> Thanks Jörg. The query does no sorting or scoring. So the child/parent
> cost must be high -- any ideas on ways to improve look ups -- would
> docvalues or upgrading to 1.4 help?
>
> Thanks
> On Saturday, February 28, 2015 at 5:46:59 AM UTC-8, Jörg Prante wrote:
>
> > No idea, except that it could be there is extensive sorting or scoring at
> > work. This would mean the whole result set has to be iterated/computed to
> > the point from where it should scroll. Also the parent/child structure
> > construction is expensive which may be not optimal for scan/scroll as you
> > already noted.
> >
> > Jörg
> >
> > On Sat, Feb 28, 2015 at 8:48 AM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com) wrote:
> >
> > > Good question. The request has several filters on child types which may
> > > have dozens to hundreds of documents per parent document.
> > >
> > > It could just be the request is quite complex at calculating the
> > > documents. The odd factor, is the query is quick at the beginning then
> > > times out after about 500K documents.
> > >
> > > The query fetches 7500 documents each go (15 shards, 500 per shard).
> > > Typical timing
> > > 2s
> > > 3s
> > > 2s
> > > ...
> > > @400K docs scrolled
> > > 10s
> > > 12s
> > > @500K docs scrolled
> > > timeout (30 seconds)
> > > ...
> > > recovers
> > > 16s -- returns 6500 results instead.
> > >
> > > Any idea on why as you scroll deeper, the response time slows any why
> > > several shards might stop returning results?
> > >
> > > On Friday, February 27, 2015 at 8:43:26 AM UTC-8, Jörg Prante wrote:
> > >
> > > > How does the scan/scroll request look like?
> > > >
> > > > Jörg
> > > >
> > > > On Fri, Feb 27, 2015 at 5:38 PM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com) wrote:
> > > >
> > > > > We have a 15 node cluster, 9 data nodes with 18gb ram, 3 master, 3
> > > > > client, 12 cpus running 1.3.2. When doing a scroll on an index of 20gb for
> > > > > 800K documents, each document is 1K we're seeing cpu jump to 30% to 90% on
> > > > > nodes with primary a shard. Index and other search quests are pretty low.
> > > > >
> > > > > Our scroll settings
> > > > > timeout: 30 seconds
> > > > > scroll size per shard: 500.
> > > > >
> > > > > -- Each scroll request returns 2500 results and takes 10-15 seconds
> > > > > per request.
> > > > >
> > > > > - iowait is 0.1%
> > > > >
> > > > > Any help appreciated. Details below:
> > > > >
> > > > > Hot threads dump
> > > > >
> > > > > 95.4% (476.9ms out of 500ms) cpu usage by thread
> > > > > 'elasticsearch[esd1][search][T#5]'
> > > > > 5/10 snapshots sharing following 37 elements
> > > > > org.apache.lucene.util.packed.MonotonicAppendingLongBuffer.get(
> > > > > MonotonicAppendingLongBuffer.java:70)
> > > > > org.apache.lucene.util.packed.AbstractAppendingLongBuffer.get(
> > > > > AbstractAppendingLongBuffer.java:102)
> > > > > org.apache.lucene.index.XOrdinalMap$2.get(XOrdinalMap.java:260)
> > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > GlobalOrdinalMapping.getGlobalOrd(GlobalOrdinalMapping.java:54)
> > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > GlobalOrdinalMapping.getOrd(GlobalOrdinalMapping.java:63)
> > > > > org.elasticsearch.index.search.child.
> > > > > ChildrenConstantScoreQuery$ParentOrdCollector.collect(
> > > > > ChildrenConstantScoreQuery.java:269)
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > scoreAll(Weight.java:193)
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > score(Weight.java:163)
> > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:621)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:309)
> > > > > org.elasticsearch.index.search.child.
> > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > java:120)
> > > > > org.apache.lucene.search.IndexSearcher.createNormalizedWeight(
> > > > > IndexSearcher.java:684)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > org.elasticsearch.index.search.child.CustomQueryWrappingFilter.
> > > > > getDocIdSet(CustomQueryWrappingFilter.java:72)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.
> > > > > getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > FilteredQuery.java:136)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > FilteredQuery.java:150)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:618)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > search(ContextIndexSearcher.java:175)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:309)
> > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > ScanContext.java:52)
> > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > QueryPhase.java:119)
> > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > SearchService.java:237)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:856)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:845)
> > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1145)
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > java.lang.Thread.run(Thread.java:744)
> > > > > 3/10 snapshots sharing following 35 elements
> > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > FilteredDocIdSetIterator.java:59)
> > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > primaryNext(FilteredQuery.java:290)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > PrimaryAdvancedLeapFrogScorer.primaryNext(FilteredQuery.java:340)
> > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > nextDoc(FilteredQuery.java:285)
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > scoreAll(Weight.java:192)
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > score(Weight.java:163)
> > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:621)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:309)
> > > > > org.elasticsearch.index.search.child.
> > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > java:120)
> > > > > org.apache.lucene.search.IndexSearcher.createNormalizedWeight(
> > > > > IndexSearcher.java:684)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > org.elasticsearch.index.search.child.CustomQueryWrappingFilter.
> > > > > getDocIdSet(CustomQueryWrappingFilter.java:72)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.
> > > > > getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > FilteredQuery.java:136)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > FilteredQuery.java:150)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:618)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > search(ContextIndexSearcher.java:175)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:309)
> > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > ScanContext.java:52)
> > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > QueryPhase.java:119)
> > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > SearchService.java:237)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:856)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:845)
> > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1145)
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > java.lang.Thread.run(Thread.java:744)
> > > > > 2/10 snapshots sharing following 30 elements
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > score(Weight.java:163)
> > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:621)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:309)
> > > > > org.elasticsearch.index.search.child.
> > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > java:120)
> > > > > org.apache.lucene.search.IndexSearcher.createNormalizedWeight(
> > > > > IndexSearcher.java:684)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > org.elasticsearch.index.search.child.CustomQueryWrappingFilter.
> > > > > getDocIdSet(CustomQueryWrappingFilter.java:72)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.
> > > > > getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > FilteredQuery.java:136)
> > > > > org.apache.lucene.search.FilteredQuery$
> > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > FilteredQuery.java:150)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:618)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > search(ContextIndexSearcher.java:175)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:309)
> > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > ScanContext.java:52)
> > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > QueryPhase.java:119)
> > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > SearchService.java:237)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:856)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:845)
> > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1145)
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > java.lang.Thread.run(Thread.java:744)
> > > > >
> > > > > 58.8% (294.1ms out of 500ms) cpu usage by thread
> > > > > 'elasticsearch[esd1][search][T#3]'
> > > > > 3/10 snapshots sharing following 19 elements
> > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > FilteredDocIdSetIterator.java:60)
> > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > ConstantScorer.nextDoc(ConstantScoreQuery.java:257)
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > scoreAll(Weight.java:192)
> > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > score(Weight.java:163)
> > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:621)
> > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > search(ContextIndexSearcher.java:175)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:581)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:533)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:510)
> > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > IndexSearcher.java:345)
> > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > QueryPhase.java:149)
> > > > > org.elasticsearch.search.SearchService.executeQueryPhase(
> > > > > SearchService.java:261)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchQueryTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:688)
> > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > SearchQueryTransportHandler.messageReceived(
> > > > > SearchServiceTransportAction.java:677)
> > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1145)
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > java.lang.Thread.run(Thread.java:744)
> > > > > 7/10 snapshots sharing following 10 elements
> > > > > sun.misc.Unsafe.park(Native Method)
> > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > java:186)
> > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > LinkedTransferQueue.java:735)
> > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > LinkedTransferQueue.java:644)
> > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > LinkedTransferQueue.java:1137)
> > > > > org.elasticsearch.common.util.concurrent.
> > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > ThreadPoolExecutor.java:1068)
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1130)
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > java.lang.Thread.run(Thread.java:744)
> > > > >
> > > > > 41.0% (204.8ms out of 500ms) cpu usage by thread
> > > > > 'elasticsearch[esd1][search][T#6]'
> > > > > 10/10 snapshots sharing following 10 elements
> > > > > sun.misc.Unsafe.park(Native Method)
> > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > java:186)
> > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > LinkedTransferQueue.java:735)
> > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > LinkedTransferQueue.java:644)
> > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > LinkedTransferQueue.java:1137)
> > > > > org.elasticsearch.common.util.concurrent.
> > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > ThreadPoolExecutor.java:1068)
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1130)
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > java.lang.Thread.run(Thread.java:744)
> > > > >
> > > > > Bigdesk graphs
> > > > >
> > > > > [https://lh5.googleusercontent.com/-czxx8fTd\_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png](https://lh5.googleusercontent.com/-czxx8fTd_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png)
> > > > > \
> > > > >
> > > > > [https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm\_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png](https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png)
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%
> > > > > [40googlegroups.com](http://40googlegroups.com)
> > > > > [https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4c31461f-4da9-4ede-b798-681e626438f0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4c31461f-4da9-4ede-b798-681e626438f0%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Graham\_Lenton](https://discuss.elastic.co/u/Graham_Lenton)
#### Post date: [March 3, 2015, 7:08am UTC](https://discuss.elastic.co/t/high-cpu-during-scroll-large-request-low-iowait/22435/7 "2015-03-03T07:08:13Z")
Oops should have said:
407,500 \* 15 \* 1 == 6,112,500
.. but you get the idea ..
G
On Tuesday, 3 March 2015 07:01:19 UTC, Graham Lenton wrote:
> Justin - also keep an eye on your client nodes - they will be doing quite
> a bit of garbage collection, which will only increase as paging gets deeper.
>
> Bear in mind that your client node has to reduce (paging\_depth \*
> number\_of\_shards \* number\_of\_indices) down to number\_of\_hits required for
> each and every request.
>
> So retrieving results 400,000 to 407,500 from a single index with 15
> shards will give the client node 400,000 \* 15 \* 1 == 6,000,000 results to
> crunch down to your 7,500. That will result in a great deal of GC.
>
> If this is the case, possible strategies to improve matters are:
>
> - keep the client heap size as small as possible to make GC happen more
> frequently - this depends on the type of requests and concurrency you have.
> - reindex with less shards if possible - experiment with a single shard
> with your data on a single node of your hardware to find how many docs
> _you_ can fit in a shard before performance hits the wall.
> - ensure your application uses all 3 client nodes to spread the load, and
> increase the number of client nodes if possible
> - have more than one index - if you can segment the query in a sensible
> way e.g. time series data
>
> G
>
> On Tuesday, 3 March 2015 02:21:37 UTC, Justin Zhu wrote:
>
> > Thanks Jörg. The query does no sorting or scoring. So the child/parent
> > cost must be high -- any ideas on ways to improve look ups -- would
> > docvalues or upgrading to 1.4 help?
> >
> > Thanks
> > On Saturday, February 28, 2015 at 5:46:59 AM UTC-8, Jörg Prante wrote:
> >
> > > No idea, except that it could be there is extensive sorting or scoring
> > > at work. This would mean the whole result set has to be iterated/computed
> > > to the point from where it should scroll. Also the parent/child structure
> > > construction is expensive which may be not optimal for scan/scroll as you
> > > already noted.
> > >
> > > Jörg
> > >
> > > On Sat, Feb 28, 2015 at 8:48 AM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com) wrote:
> > >
> > > > Good question. The request has several filters on child types which may
> > > > have dozens to hundreds of documents per parent document.
> > > >
> > > > It could just be the request is quite complex at calculating the
> > > > documents. The odd factor, is the query is quick at the beginning then
> > > > times out after about 500K documents.
> > > >
> > > > The query fetches 7500 documents each go (15 shards, 500 per shard).
> > > > Typical timing
> > > > 2s
> > > > 3s
> > > > 2s
> > > > ...
> > > > @400K docs scrolled
> > > > 10s
> > > > 12s
> > > > @500K docs scrolled
> > > > timeout (30 seconds)
> > > > ...
> > > > recovers
> > > > 16s -- returns 6500 results instead.
> > > >
> > > > Any idea on why as you scroll deeper, the response time slows any why
> > > > several shards might stop returning results?
> > > >
> > > > On Friday, February 27, 2015 at 8:43:26 AM UTC-8, Jörg Prante wrote:
> > > >
> > > > > How does the scan/scroll request look like?
> > > > >
> > > > > Jörg
> > > > >
> > > > > On Fri, Feb 27, 2015 at 5:38 PM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > We have a 15 node cluster, 9 data nodes with 18gb ram, 3 master, 3
> > > > > > client, 12 cpus running 1.3.2. When doing a scroll on an index of 20gb for
> > > > > > 800K documents, each document is 1K we're seeing cpu jump to 30% to 90% on
> > > > > > nodes with primary a shard. Index and other search quests are pretty low.
> > > > > >
> > > > > > Our scroll settings
> > > > > > timeout: 30 seconds
> > > > > > scroll size per shard: 500.
> > > > > >
> > > > > > -- Each scroll request returns 2500 results and takes 10-15 seconds
> > > > > > per request.
> > > > > >
> > > > > > - iowait is 0.1%
> > > > > >
> > > > > > Any help appreciated. Details below:
> > > > > >
> > > > > > Hot threads dump
> > > > > >
> > > > > > 95.4% (476.9ms out of 500ms) cpu usage by thread
> > > > > > 'elasticsearch[esd1][search][T#5]'
> > > > > > 5/10 snapshots sharing following 37 elements
> > > > > > org.apache.lucene.util.packed.MonotonicAppendingLongBuffer.
> > > > > > get(MonotonicAppendingLongBuffer.java:70)
> > > > > > org.apache.lucene.util.packed.AbstractAppendingLongBuffer.get(
> > > > > > AbstractAppendingLongBuffer.java:102)
> > > > > > org.apache.lucene.index.XOrdinalMap$2.get(XOrdinalMap.
> > > > > > java:260)
> > > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > > GlobalOrdinalMapping.getGlobalOrd(GlobalOrdinalMapping.java:54)
> > > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > > GlobalOrdinalMapping.getOrd(GlobalOrdinalMapping.java:63)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > ChildrenConstantScoreQuery$ParentOrdCollector.collect(
> > > > > > ChildrenConstantScoreQuery.java:269)
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > scoreAll(Weight.java:193)
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > score(Weight.java:163)
> > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:621)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:309)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > java:120)
> > > > > > org.apache.lucene.search.IndexSearcher.createNormalizedWeight(
> > > > > > IndexSearcher.java:684)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > java:72)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.
> > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > FilteredQuery.java:136)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > FilteredQuery.java:150)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:618)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > search(ContextIndexSearcher.java:175)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:309)
> > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > ScanContext.java:52)
> > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > QueryPhase.java:119)
> > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > SearchService.java:237)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:856)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:845)
> > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1145)
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:615)
> > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > 3/10 snapshots sharing following 35 elements
> > > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > > FilteredDocIdSetIterator.java:59)
> > > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > > primaryNext(FilteredQuery.java:290)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > PrimaryAdvancedLeapFrogScorer.primaryNext(FilteredQuery.java:340)
> > > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > > nextDoc(FilteredQuery.java:285)
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > scoreAll(Weight.java:192)
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > score(Weight.java:163)
> > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:621)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:309)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > java:120)
> > > > > > org.apache.lucene.search.IndexSearcher.createNormalizedWeight(
> > > > > > IndexSearcher.java:684)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > java:72)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.
> > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > FilteredQuery.java:136)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > FilteredQuery.java:150)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:618)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > search(ContextIndexSearcher.java:175)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:309)
> > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > ScanContext.java:52)
> > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > QueryPhase.java:119)
> > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > SearchService.java:237)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:856)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:845)
> > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1145)
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:615)
> > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > 2/10 snapshots sharing following 30 elements
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > score(Weight.java:163)
> > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:621)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:309)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > java:120)
> > > > > > org.apache.lucene.search.IndexSearcher.createNormalizedWeight(
> > > > > > IndexSearcher.java:684)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > org.elasticsearch.index.search.child.
> > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > java:72)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > org.elasticsearch.common.lucene.search.
> > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > FilteredQuery.java:136)
> > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > FilteredQuery.java:150)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:618)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > search(ContextIndexSearcher.java:175)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:309)
> > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > ScanContext.java:52)
> > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > QueryPhase.java:119)
> > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > SearchService.java:237)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:856)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:845)
> > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1145)
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:615)
> > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > >
> > > > > > 58.8% (294.1ms out of 500ms) cpu usage by thread
> > > > > > 'elasticsearch[esd1][search][T#3]'
> > > > > > 3/10 snapshots sharing following 19 elements
> > > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > > FilteredDocIdSetIterator.java:60)
> > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > ConstantScorer.nextDoc(ConstantScoreQuery.java:257)
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > scoreAll(Weight.java:192)
> > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > score(Weight.java:163)
> > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:621)
> > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > search(ContextIndexSearcher.java:175)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:581)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:533)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:510)
> > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > IndexSearcher.java:345)
> > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > QueryPhase.java:149)
> > > > > > org.elasticsearch.search.SearchService.executeQueryPhase(
> > > > > > SearchService.java:261)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchQueryTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:688)
> > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > SearchQueryTransportHandler.messageReceived(
> > > > > > SearchServiceTransportAction.java:677)
> > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1145)
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:615)
> > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > 7/10 snapshots sharing following 10 elements
> > > > > > sun.misc.Unsafe.park(Native Method)
> > > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > > java:186)
> > > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > > LinkedTransferQueue.java:735)
> > > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > > LinkedTransferQueue.java:644)
> > > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > > LinkedTransferQueue.java:1137)
> > > > > > org.elasticsearch.common.util.concurrent.
> > > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > > ThreadPoolExecutor.java:1068)
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1130)
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:615)
> > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > >
> > > > > > 41.0% (204.8ms out of 500ms) cpu usage by thread
> > > > > > 'elasticsearch[esd1][search][T#6]'
> > > > > > 10/10 snapshots sharing following 10 elements
> > > > > > sun.misc.Unsafe.park(Native Method)
> > > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > > java:186)
> > > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > > LinkedTransferQueue.java:735)
> > > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > > LinkedTransferQueue.java:644)
> > > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > > LinkedTransferQueue.java:1137)
> > > > > > org.elasticsearch.common.util.concurrent.
> > > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > > ThreadPoolExecutor.java:1068)
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1130)
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:615)
> > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > >
> > > > > > Bigdesk graphs
> > > > > >
> > > > > > [https://lh5.googleusercontent.com/-czxx8fTd\_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png](https://lh5.googleusercontent.com/-czxx8fTd_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png)
> > > > > > \
> > > > > >
> > > > > > [https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm\_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png](https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png)
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > > msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%
> > > > > > [40googlegroups.com](http://40googlegroups.com)
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > .
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com)
> > > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > .
> > > >
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/50e87bcc-30d1-47cf-9118-40002354725f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/50e87bcc-30d1-47cf-9118-40002354725f%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Justin\_Zhu\_2](https://discuss.elastic.co/u/Justin_Zhu_2)
#### Post date: [March 3, 2015, 5:55pm UTC](https://discuss.elastic.co/t/high-cpu-during-scroll-large-request-low-iowait/22435/8 "2015-03-03T17:55:54Z")
Graham, thanks for the info. Just to confirm, is this what happens for a
scroll request and not a paged search?
On Monday, March 2, 2015 at 11:08:13 PM UTC-8, Graham Lenton wrote:
> Oops should have said:
>
> 407,500 \* 15 \* 1 == 6,112,500
>
> .. but you get the idea ..
>
> G
>
> On Tuesday, 3 March 2015 07:01:19 UTC, Graham Lenton wrote:
>
> > Justin - also keep an eye on your client nodes - they will be doing quite
> > a bit of garbage collection, which will only increase as paging gets deeper.
> >
> > Bear in mind that your client node has to reduce (paging\_depth \*
> > number\_of\_shards \* number\_of\_indices) down to number\_of\_hits required for
> > each and every request.
> >
> > So retrieving results 400,000 to 407,500 from a single index with 15
> > shards will give the client node 400,000 \* 15 \* 1 == 6,000,000 results to
> > crunch down to your 7,500. That will result in a great deal of GC.
> >
> > If this is the case, possible strategies to improve matters are:
> >
> > - keep the client heap size as small as possible to make GC happen more
> > frequently - this depends on the type of requests and concurrency you have.
> > - reindex with less shards if possible - experiment with a single shard
> > with your data on a single node of your hardware to find how many docs
> > _you_ can fit in a shard before performance hits the wall.
> > - ensure your application uses all 3 client nodes to spread the load,
> > and increase the number of client nodes if possible
> > - have more than one index - if you can segment the query in a sensible
> > way e.g. time series data
> >
> > G
> >
> > On Tuesday, 3 March 2015 02:21:37 UTC, Justin Zhu wrote:
> >
> > > Thanks Jörg. The query does no sorting or scoring. So the child/parent
> > > cost must be high -- any ideas on ways to improve look ups -- would
> > > docvalues or upgrading to 1.4 help?
> > >
> > > Thanks
> > > On Saturday, February 28, 2015 at 5:46:59 AM UTC-8, Jörg Prante wrote:
> > >
> > > > No idea, except that it could be there is extensive sorting or scoring
> > > > at work. This would mean the whole result set has to be iterated/computed
> > > > to the point from where it should scroll. Also the parent/child structure
> > > > construction is expensive which may be not optimal for scan/scroll as you
> > > > already noted.
> > > >
> > > > Jörg
> > > >
> > > > On Sat, Feb 28, 2015 at 8:48 AM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com) wrote:
> > > >
> > > > > Good question. The request has several filters on child types which
> > > > > may have dozens to hundreds of documents per parent document.
> > > > >
> > > > > It could just be the request is quite complex at calculating the
> > > > > documents. The odd factor, is the query is quick at the beginning then
> > > > > times out after about 500K documents.
> > > > >
> > > > > The query fetches 7500 documents each go (15 shards, 500 per shard).
> > > > > Typical timing
> > > > > 2s
> > > > > 3s
> > > > > 2s
> > > > > ...
> > > > > @400K docs scrolled
> > > > > 10s
> > > > > 12s
> > > > > @500K docs scrolled
> > > > > timeout (30 seconds)
> > > > > ...
> > > > > recovers
> > > > > 16s -- returns 6500 results instead.
> > > > >
> > > > > Any idea on why as you scroll deeper, the response time slows any why
> > > > > several shards might stop returning results?
> > > > >
> > > > > On Friday, February 27, 2015 at 8:43:26 AM UTC-8, Jörg Prante wrote:
> > > > >
> > > > > > How does the scan/scroll request look like?
> > > > > >
> > > > > > Jörg
> > > > > >
> > > > > > On Fri, Feb 27, 2015 at 5:38 PM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com)
> > > > > > wrote:
> > > > > >
> > > > > > > We have a 15 node cluster, 9 data nodes with 18gb ram, 3 master, 3
> > > > > > > client, 12 cpus running 1.3.2. When doing a scroll on an index of 20gb for
> > > > > > > 800K documents, each document is 1K we're seeing cpu jump to 30% to 90% on
> > > > > > > nodes with primary a shard. Index and other search quests are pretty low.
> > > > > > >
> > > > > > > Our scroll settings
> > > > > > > timeout: 30 seconds
> > > > > > > scroll size per shard: 500.
> > > > > > >
> > > > > > > -- Each scroll request returns 2500 results and takes 10-15 seconds
> > > > > > > per request.
> > > > > > >
> > > > > > > - iowait is 0.1%
> > > > > > >
> > > > > > > Any help appreciated. Details below:
> > > > > > >
> > > > > > > Hot threads dump
> > > > > > >
> > > > > > > 95.4% (476.9ms out of 500ms) cpu usage by thread
> > > > > > > 'elasticsearch[esd1][search][T#5]'
> > > > > > > 5/10 snapshots sharing following 37 elements
> > > > > > > org.apache.lucene.util.packed.MonotonicAppendingLongBuffer.
> > > > > > > get(MonotonicAppendingLongBuffer.java:70)
> > > > > > > org.apache.lucene.util.packed.AbstractAppendingLongBuffer.
> > > > > > > get(AbstractAppendingLongBuffer.java:102)
> > > > > > > org.apache.lucene.index.XOrdinalMap$2.get(XOrdinalMap.
> > > > > > > java:260)
> > > > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > > > GlobalOrdinalMapping.getGlobalOrd(GlobalOrdinalMapping.java:54)
> > > > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > > > GlobalOrdinalMapping.getOrd(GlobalOrdinalMapping.java:63)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > ChildrenConstantScoreQuery$ParentOrdCollector.collect(
> > > > > > > ChildrenConstantScoreQuery.java:269)
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > scoreAll(Weight.java:193)
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > score(Weight.java:163)
> > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:621)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:309)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > > java:120)
> > > > > > > org.apache.lucene.search.IndexSearcher.
> > > > > > > createNormalizedWeight(IndexSearcher.java:684)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > > java:72)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.
> > > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > > FilteredQuery.java:136)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > > FilteredQuery.java:150)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:618)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:309)
> > > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > > ScanContext.java:52)
> > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > QueryPhase.java:119)
> > > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > > SearchService.java:237)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:856)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:845)
> > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > 3/10 snapshots sharing following 35 elements
> > > > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > > > FilteredDocIdSetIterator.java:59)
> > > > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > > > primaryNext(FilteredQuery.java:290)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > PrimaryAdvancedLeapFrogScorer.primaryNext(FilteredQuery.java:340)
> > > > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > > > nextDoc(FilteredQuery.java:285)
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > scoreAll(Weight.java:192)
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > score(Weight.java:163)
> > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:621)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:309)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > > java:120)
> > > > > > > org.apache.lucene.search.IndexSearcher.
> > > > > > > createNormalizedWeight(IndexSearcher.java:684)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > > java:72)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.
> > > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > > FilteredQuery.java:136)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > > FilteredQuery.java:150)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:618)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:309)
> > > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > > ScanContext.java:52)
> > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > QueryPhase.java:119)
> > > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > > SearchService.java:237)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:856)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:845)
> > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > 2/10 snapshots sharing following 30 elements
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > score(Weight.java:163)
> > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:621)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:309)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > > java:120)
> > > > > > > org.apache.lucene.search.IndexSearcher.
> > > > > > > createNormalizedWeight(IndexSearcher.java:684)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > > org.elasticsearch.index.search.child.
> > > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > > java:72)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > org.elasticsearch.common.lucene.search.
> > > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > > FilteredQuery.java:136)
> > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > > FilteredQuery.java:150)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:618)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:309)
> > > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > > ScanContext.java:52)
> > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > QueryPhase.java:119)
> > > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > > SearchService.java:237)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:856)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchScanScrollTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:845)
> > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > >
> > > > > > > 58.8% (294.1ms out of 500ms) cpu usage by thread
> > > > > > > 'elasticsearch[esd1][search][T#3]'
> > > > > > > 3/10 snapshots sharing following 19 elements
> > > > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > > > FilteredDocIdSetIterator.java:60)
> > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > ConstantScorer.nextDoc(ConstantScoreQuery.java:257)
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > scoreAll(Weight.java:192)
> > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > score(Weight.java:163)
> > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:621)
> > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:581)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:533)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:510)
> > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > IndexSearcher.java:345)
> > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > QueryPhase.java:149)
> > > > > > > org.elasticsearch.search.SearchService.executeQueryPhase(
> > > > > > > SearchService.java:261)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchQueryTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:688)
> > > > > > > org.elasticsearch.search.action.SearchServiceTransportAction$
> > > > > > > SearchQueryTransportHandler.messageReceived(
> > > > > > > SearchServiceTransportAction.java:677)
> > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > 7/10 snapshots sharing following 10 elements
> > > > > > > sun.misc.Unsafe.park(Native Method)
> > > > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > > > java:186)
> > > > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > > > LinkedTransferQueue.java:735)
> > > > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > > > LinkedTransferQueue.java:644)
> > > > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > > > LinkedTransferQueue.java:1137)
> > > > > > > org.elasticsearch.common.util.concurrent.
> > > > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > > > ThreadPoolExecutor.java:1068)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1130)
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > >
> > > > > > > 41.0% (204.8ms out of 500ms) cpu usage by thread
> > > > > > > 'elasticsearch[esd1][search][T#6]'
> > > > > > > 10/10 snapshots sharing following 10 elements
> > > > > > > sun.misc.Unsafe.park(Native Method)
> > > > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > > > java:186)
> > > > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > > > LinkedTransferQueue.java:735)
> > > > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > > > LinkedTransferQueue.java:644)
> > > > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > > > LinkedTransferQueue.java:1137)
> > > > > > > org.elasticsearch.common.util.concurrent.
> > > > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > > > ThreadPoolExecutor.java:1068)
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1130)
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > >
> > > > > > > Bigdesk graphs
> > > > > > >
> > > > > > > [https://lh5.googleusercontent.com/-czxx8fTd\_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png](https://lh5.googleusercontent.com/-czxx8fTd_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png)
> > > > > > > \
> > > > > > >
> > > > > > > [https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm\_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png](https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png)
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > To view this discussion on the web visit
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/72fb97ee-](https://groups.google.com/d/msgid/elasticsearch/72fb97ee-)
> > > > > > > a02f-48e3-b217-cd3e15cc24e3%[40googlegroups.com](http://40googlegroups.com)
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > > .
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com)
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > .
> > > > >
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cf5e70d5-3ff4-4375-bda8-9d2161730f66%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cf5e70d5-3ff4-4375-bda8-9d2161730f66%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Graham\_Lenton](https://discuss.elastic.co/u/Graham_Lenton)
#### Post date: [March 3, 2015, 11:11pm UTC](https://discuss.elastic.co/t/high-cpu-during-scroll-large-request-low-iowait/22435/9 "2015-03-03T23:11:24Z")
Justin to be honest I am not 100% sure and don't have time to check the
source code, but I believe the main difference with scroll requests is that
the data nodes keep some resources around and keep track of which documents
have been returned. It's not clear from the documentation what the
interaction with the client nodes is in this case. I'm sure someone more
knowledgeable can chime in here.
On Tuesday, 3 March 2015 17:55:54 UTC, Justin Zhu wrote:
> Graham, thanks for the info. Just to confirm, is this what happens for a
> scroll request and not a paged search?
>
> On Monday, March 2, 2015 at 11:08:13 PM UTC-8, Graham Lenton wrote:
>
> > Oops should have said:
> >
> > 407,500 \* 15 \* 1 == 6,112,500
> >
> > .. but you get the idea ..
> >
> > G
> >
> > On Tuesday, 3 March 2015 07:01:19 UTC, Graham Lenton wrote:
> >
> > > Justin - also keep an eye on your client nodes - they will be doing
> > > quite a bit of garbage collection, which will only increase as paging gets
> > > deeper.
> > >
> > > Bear in mind that your client node has to reduce (paging\_depth \*
> > > number\_of\_shards \* number\_of\_indices) down to number\_of\_hits required for
> > > each and every request.
> > >
> > > So retrieving results 400,000 to 407,500 from a single index with 15
> > > shards will give the client node 400,000 \* 15 \* 1 == 6,000,000 results to
> > > crunch down to your 7,500. That will result in a great deal of GC.
> > >
> > > If this is the case, possible strategies to improve matters are:
> > >
> > > - keep the client heap size as small as possible to make GC happen more
> > > frequently - this depends on the type of requests and concurrency you have.
> > > - reindex with less shards if possible - experiment with a single shard
> > > with your data on a single node of your hardware to find how many docs
> > > _you_ can fit in a shard before performance hits the wall.
> > > - ensure your application uses all 3 client nodes to spread the load,
> > > and increase the number of client nodes if possible
> > > - have more than one index - if you can segment the query in a sensible
> > > way e.g. time series data
> > >
> > > G
> > >
> > > On Tuesday, 3 March 2015 02:21:37 UTC, Justin Zhu wrote:
> > >
> > > > Thanks Jörg. The query does no sorting or scoring. So the child/parent
> > > > cost must be high -- any ideas on ways to improve look ups -- would
> > > > docvalues or upgrading to 1.4 help?
> > > >
> > > > Thanks
> > > > On Saturday, February 28, 2015 at 5:46:59 AM UTC-8, Jörg Prante wrote:
> > > >
> > > > > No idea, except that it could be there is extensive sorting or scoring
> > > > > at work. This would mean the whole result set has to be iterated/computed
> > > > > to the point from where it should scroll. Also the parent/child structure
> > > > > construction is expensive which may be not optimal for scan/scroll as you
> > > > > already noted.
> > > > >
> > > > > Jörg
> > > > >
> > > > > On Sat, Feb 28, 2015 at 8:48 AM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Good question. The request has several filters on child types which
> > > > > > may have dozens to hundreds of documents per parent document.
> > > > > >
> > > > > > It could just be the request is quite complex at calculating the
> > > > > > documents. The odd factor, is the query is quick at the beginning then
> > > > > > times out after about 500K documents.
> > > > > >
> > > > > > The query fetches 7500 documents each go (15 shards, 500 per shard).
> > > > > > Typical timing
> > > > > > 2s
> > > > > > 3s
> > > > > > 2s
> > > > > > ...
> > > > > > @400K docs scrolled
> > > > > > 10s
> > > > > > 12s
> > > > > > @500K docs scrolled
> > > > > > timeout (30 seconds)
> > > > > > ...
> > > > > > recovers
> > > > > > 16s -- returns 6500 results instead.
> > > > > >
> > > > > > Any idea on why as you scroll deeper, the response time slows any why
> > > > > > several shards might stop returning results?
> > > > > >
> > > > > > On Friday, February 27, 2015 at 8:43:26 AM UTC-8, Jörg Prante wrote:
> > > > > >
> > > > > > > How does the scan/scroll request look like?
> > > > > > >
> > > > > > > Jörg
> > > > > > >
> > > > > > > On Fri, Feb 27, 2015 at 5:38 PM, Justin Zhu [haora...@gmail.com](mailto:haora...@gmail.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We have a 15 node cluster, 9 data nodes with 18gb ram, 3 master, 3
> > > > > > > > client, 12 cpus running 1.3.2. When doing a scroll on an index of 20gb for
> > > > > > > > 800K documents, each document is 1K we're seeing cpu jump to 30% to 90% on
> > > > > > > > nodes with primary a shard. Index and other search quests are pretty low.
> > > > > > > >
> > > > > > > > Our scroll settings
> > > > > > > > timeout: 30 seconds
> > > > > > > > scroll size per shard: 500.
> > > > > > > >
> > > > > > > > -- Each scroll request returns 2500 results and takes 10-15 seconds
> > > > > > > > per request.
> > > > > > > >
> > > > > > > > - iowait is 0.1%
> > > > > > > >
> > > > > > > > Any help appreciated. Details below:
> > > > > > > >
> > > > > > > > Hot threads dump
> > > > > > > >
> > > > > > > > 95.4% (476.9ms out of 500ms) cpu usage by thread
> > > > > > > > 'elasticsearch[esd1][search][T#5]'
> > > > > > > > 5/10 snapshots sharing following 37 elements
> > > > > > > > org.apache.lucene.util.packed.MonotonicAppendingLongBuffer.
> > > > > > > > get(MonotonicAppendingLongBuffer.java:70)
> > > > > > > > org.apache.lucene.util.packed.AbstractAppendingLongBuffer.
> > > > > > > > get(AbstractAppendingLongBuffer.java:102)
> > > > > > > > org.apache.lucene.index.XOrdinalMap$2.get(XOrdinalMap.
> > > > > > > > java:260)
> > > > > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > > > > GlobalOrdinalMapping.getGlobalOrd(GlobalOrdinalMapping.java:54)
> > > > > > > > org.elasticsearch.index.fielddata.ordinals.
> > > > > > > > GlobalOrdinalMapping.getOrd(GlobalOrdinalMapping.java:63)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > ChildrenConstantScoreQuery$ParentOrdCollector.collect(
> > > > > > > > ChildrenConstantScoreQuery.java:269)
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > scoreAll(Weight.java:193)
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > score(Weight.java:163)
> > > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.
> > > > > > > > java:35)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:621)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:309)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > > > java:120)
> > > > > > > > org.apache.lucene.search.IndexSearcher.
> > > > > > > > createNormalizedWeight(IndexSearcher.java:684)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > > > java:72)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.
> > > > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:
> > > > > > > > 46)
> > > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > > > FilteredQuery.java:136)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > > > FilteredQuery.java:150)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:618)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:309)
> > > > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > > > ScanContext.java:52)
> > > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > > QueryPhase.java:119)
> > > > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > > > SearchService.java:237)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchScanScrollTransportHandl
> > > > > > > > er.messageReceived(SearchServiceTransportAction.java:856)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchScanScrollTransportHandl
> > > > > > > > er.messageReceived(SearchServiceTransportAction.java:845)
> > > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > > 3/10 snapshots sharing following 35 elements
> > > > > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > > > > FilteredDocIdSetIterator.java:59)
> > > > > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > > > > primaryNext(FilteredQuery.java:290)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > PrimaryAdvancedLeapFrogScorer.primaryNext(FilteredQuery.java:340)
> > > > > > > > org.apache.lucene.search.FilteredQuery$LeapFrogScorer.
> > > > > > > > nextDoc(FilteredQuery.java:285)
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > scoreAll(Weight.java:192)
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > score(Weight.java:163)
> > > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.
> > > > > > > > java:35)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:621)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:309)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > > > java:120)
> > > > > > > > org.apache.lucene.search.IndexSearcher.
> > > > > > > > createNormalizedWeight(IndexSearcher.java:684)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > > > java:72)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.
> > > > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:
> > > > > > > > 46)
> > > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > > > FilteredQuery.java:136)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > > > FilteredQuery.java:150)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:618)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:309)
> > > > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > > > ScanContext.java:52)
> > > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > > QueryPhase.java:119)
> > > > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > > > SearchService.java:237)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchScanScrollTransportHandl
> > > > > > > > er.messageReceived(SearchServiceTransportAction.java:856)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchScanScrollTransportHandl
> > > > > > > > er.messageReceived(SearchServiceTransportAction.java:845)
> > > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > > 2/10 snapshots sharing following 30 elements
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > score(Weight.java:163)
> > > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.
> > > > > > > > java:35)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:621)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:309)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.
> > > > > > > > java:120)
> > > > > > > > org.apache.lucene.search.IndexSearcher.
> > > > > > > > createNormalizedWeight(IndexSearcher.java:684)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > createNormalizedWeight(ContextIndexSearcher.java:132)
> > > > > > > > org.elasticsearch.index.search.child.
> > > > > > > > CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.
> > > > > > > > java:72)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.XBooleanFilter.
> > > > > > > > getDocIdSet(XBooleanFilter.java:83)
> > > > > > > > org.elasticsearch.common.lucene.search.
> > > > > > > > ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:
> > > > > > > > 46)
> > > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > > ConstantWeight.scorer(ConstantScoreQuery.java:157)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547)
> > > > > > > > org.apache.lucene.search.FilteredQuery$1.scorer(
> > > > > > > > FilteredQuery.java:136)
> > > > > > > > org.apache.lucene.search.FilteredQuery$
> > > > > > > > RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> > > > > > > > org.apache.lucene.search.FilteredQuery$FilterStrategy.
> > > > > > > > filteredBulkScorer(FilteredQuery.java:504)
> > > > > > > > org.apache.lucene.search.FilteredQuery$1.bulkScorer(
> > > > > > > > FilteredQuery.java:150)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:618)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:309)
> > > > > > > > org.elasticsearch.search.scan.ScanContext.execute(
> > > > > > > > ScanContext.java:52)
> > > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > > QueryPhase.java:119)
> > > > > > > > org.elasticsearch.search.SearchService.executeScan(
> > > > > > > > SearchService.java:237)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchScanScrollTransportHandl
> > > > > > > > er.messageReceived(SearchServiceTransportAction.java:856)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchScanScrollTransportHandl
> > > > > > > > er.messageReceived(SearchServiceTransportAction.java:845)
> > > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > >
> > > > > > > > 58.8% (294.1ms out of 500ms) cpu usage by thread
> > > > > > > > 'elasticsearch[esd1][search][T#3]'
> > > > > > > > 3/10 snapshots sharing following 19 elements
> > > > > > > > org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(
> > > > > > > > FilteredDocIdSetIterator.java:60)
> > > > > > > > org.apache.lucene.search.ConstantScoreQuery$
> > > > > > > > ConstantScorer.nextDoc(ConstantScoreQuery.java:257)
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > scoreAll(Weight.java:192)
> > > > > > > > org.apache.lucene.search.Weight$DefaultBulkScorer.
> > > > > > > > score(Weight.java:163)
> > > > > > > > org.apache.lucene.search.BulkScorer.score(BulkScorer.
> > > > > > > > java:35)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:621)
> > > > > > > > org.elasticsearch.search.internal.ContextIndexSearcher.
> > > > > > > > search(ContextIndexSearcher.java:175)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:581)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:533)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:510)
> > > > > > > > org.apache.lucene.search.IndexSearcher.search(
> > > > > > > > IndexSearcher.java:345)
> > > > > > > > org.elasticsearch.search.query.QueryPhase.execute(
> > > > > > > > QueryPhase.java:149)
> > > > > > > > org.elasticsearch.search.SearchService.executeQueryPhase(
> > > > > > > > SearchService.java:261)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchQueryTransportHandler.
> > > > > > > > messageReceived(SearchServiceTransportAction.java:688)
> > > > > > > > org.elasticsearch.search.action.
> > > > > > > > SearchServiceTransportAction$SearchQueryTransportHandler.
> > > > > > > > messageReceived(SearchServiceTransportAction.java:677)
> > > > > > > > org.elasticsearch.transport.netty.MessageChannelHandler$
> > > > > > > > RequestHandler.run(MessageChannelHandler.java:275)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1145)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > > 7/10 snapshots sharing following 10 elements
> > > > > > > > sun.misc.Unsafe.park(Native Method)
> > > > > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > > > > java:186)
> > > > > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > > > > LinkedTransferQueue.java:735)
> > > > > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > > > > LinkedTransferQueue.java:644)
> > > > > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > > > > LinkedTransferQueue.java:1137)
> > > > > > > > org.elasticsearch.common.util.concurrent.
> > > > > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > > > > ThreadPoolExecutor.java:1068)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1130)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > >
> > > > > > > > 41.0% (204.8ms out of 500ms) cpu usage by thread
> > > > > > > > 'elasticsearch[esd1][search][T#6]'
> > > > > > > > 10/10 snapshots sharing following 10 elements
> > > > > > > > sun.misc.Unsafe.park(Native Method)
> > > > > > > > java.util.concurrent.locks.LockSupport.park(LockSupport.
> > > > > > > > java:186)
> > > > > > > > java.util.concurrent.LinkedTransferQueue.awaitMatch(
> > > > > > > > LinkedTransferQueue.java:735)
> > > > > > > > java.util.concurrent.LinkedTransferQueue.xfer(
> > > > > > > > LinkedTransferQueue.java:644)
> > > > > > > > java.util.concurrent.LinkedTransferQueue.take(
> > > > > > > > LinkedTransferQueue.java:1137)
> > > > > > > > org.elasticsearch.common.util.concurrent.
> > > > > > > > SizeBlockingQueue.take(SizeBlockingQueue.java:162)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(
> > > > > > > > ThreadPoolExecutor.java:1068)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1130)
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:615)
> > > > > > > > java.lang.Thread.run(Thread.java:744)
> > > > > > > >
> > > > > > > > Bigdesk graphs
> > > > > > > >
> > > > > > > > [https://lh5.googleusercontent.com/-czxx8fTd\_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png](https://lh5.googleusercontent.com/-czxx8fTd_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png)
> > > > > > > > \
> > > > > > > >
> > > > > > > > [https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm\_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png](https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png)
> > > > > > > >
> > > > > > > > --
> > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > Groups "elasticsearch" group.
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > > To view this discussion on the web visit
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/72fb97ee-](https://groups.google.com/d/msgid/elasticsearch/72fb97ee-)
> > > > > > > > a02f-48e3-b217-cd3e15cc24e3%[40googlegroups.com](http://40googlegroups.com)
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > > > .
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > To view this discussion on the web visit
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com)
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > > .
> > > > > >
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eb0322fb-64ec-4df5-a49d-e5635cc39fc7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eb0322fb-64ec-4df5-a49d-e5635cc39fc7%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:28am UTC](https://discuss.elastic.co/t/high-cpu-during-scroll-large-request-low-iowait/22435/10 "2017-07-06T00:28:47Z")