# BucketCollector takes long time occasionally **URL:** \ **Category:** Elasticsearch\ **Created:** [July 14, 2016, 5:09am UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468 "2016-07-14T05:09:48Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [July 14, 2016, 5:09am UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468/1 "2016-07-14T05:09:48Z")
Hi. I execute a aggregation query which matches only less than 100 documents over the index which has 200m documents with 10 shards (1 replica) Most of the time, the query is executed very quickly (less than 20ms) but sometimes it takes 300ms ~ a couple of seconds. I cannot figure out which one is a problem. Before profiling, I thought Disk I/O would be the problem. But It turned out that `BucketCollector` took long time at that time. As you can see below, all queries are ended within 10ms, but `BucketCollector` was took 286ms (As I said, Collector would collect only less than 100 documents) **My Questions** 1. The result of profiling is trustworthy? (sometimes `time` is greater than `took`) 2. If `BucketCollect` is the root cause of lag, what can I do for improving performance? I hope you can give me recommendations. Thanks in advanced. ```auto "profile": { "shards": [ { "id": "[...][index-name][4]", "searches": [ { "query": [ { "query_type": "ConstantScoreQuery", "lucene": "ConstantScore(+field1:keyword1 +field2:keyword2 +(+*:* -field3:) +ConstantScore(_field_names:field3) +(+*:* -field4) +ConstantScore(_field_names:field4))", "time": "8.855492000ms", ... }, "children": [ { "query_type": "BooleanQuery", "lucene": "+field1:keyword1 +field2:keyword2 +(+*:* -field3:) +ConstantScore(_field_names:field3) +(+*:* -field4) +ConstantScore(_field_names:field4)", "time": "6.159800000ms", ... "children": [ { "query_type": "TermQuery", "lucene": "field1:keyword1", "time": "0.3522230000ms", ... }, { "query_type": "TermQuery", "lucene": "field2:keyword2", "time": "0.8661380000ms", ... }, { "query_type": "BooleanQuery", "lucene": "+*:* -field3:", "time": "0.5188010000ms", ... "children": [ { "query_type": "MatchAllDocsQuery", "lucene": "*:*", "time": "0.0008740000000ms", ... }, { "query_type": "TermQuery", "lucene": "field3:", "time": "0.1813230000ms", ... } ] }, ... ... ] } ] } ], "rewrite_time": 22178, "collector": [ { "name": "MultiCollector", "reason": "search_multi", "time": "286.5113430ms", "children": [ { "name": "TotalHitCountCollector", "reason": "search_count", "time": "0.01601100000ms" }, { "name": "BucketCollector: [[agg1, agg1_cnt]]", "reason": "aggregation", "time": "286.4594230ms" } ] } ] } ] } ``` ---
**Author:** ![jettro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jettro/32/3418_2.png) [@jettro](https://discuss.elastic.co/u/jettro)\ **Post date:** [July 14, 2016, 6:22am UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468/2 "2016-07-14T06:22:47Z")
Maybe provide a bit more information on the versions that you use, the amount of nodes and available memory. Could it be you run into memory issues? DO you allocate all memory at once? Did you check for Garbage Collect runs? Maybe system itself is out of memory? ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [July 14, 2016, 6:49am UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468/3 "2016-07-14T06:49:16Z")
Sorry for insufficient information. - ES 2.3.2 - more than 10 nodes - no gc occurred at that time (no young and old gc, I'm using CMS gc) - data size is greater than memory - _DO you allocate all memory at once?_ =\> No, I can't - `executionHint: map` - `collect_mode: breadth_first` As you expect, Disk I/O is suspicious, if then I think `searches` should be slow. I'm wondering why `BucketCollector` is slow. I think `BucketCollector` is a CPU bound job because it aggregates filtered documents from disk. Can `BucketCollector` generate Disk I/O? Thanks. ---
**Author:** ![jettro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jettro/32/3418_2.png) [@jettro](https://discuss.elastic.co/u/jettro)\ **Post date:** [July 14, 2016, 7:49am UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468/4 "2016-07-14T07:49:05Z")
Aggregations use doc\_values, in the end these are files on disk put into the OS memory if I am correct. So it could be Disk I/O ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [July 14, 2016, 8:25am UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468/5 "2016-07-14T08:25:41Z")
> Aggregations use doc\_values, in the end these are files on disk put into the OS memory if I am correct. So it could be Disk I/O That makes sense. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:35pm UTC](https://discuss.elastic.co/t/bucketcollector-takes-long-time-occasionally/55468/6 "2017-07-05T22:35:47Z")