# Aggregation is skipping needed data and not give expected results **URL:** \ **Category:** Elasticsearch\ **Created:** [March 20, 2018, 6:29pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837 "2018-03-20T18:29:07Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 20, 2018, 6:29pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/1 "2018-03-20T18:29:07Z")
Hello guys. When i'm using the following aggregation: ```auto "2": { "terms": { "field": "name", "size": 100, "order": { "1": "desc" } }, "aggs": { "1": { "avg": { "field": "value" } }, "3": { "date_histogram": { "field": "@timestamp", "interval": "1d", "time_zone": "UTC", "min_doc_count": 1 }, "aggs": { "1": { "avg": { "field": "value" } } } } } } } ``` I'm facing with situation when i have missing data blocks on the chart, because in some indices highlighted block is not present in this top 100 ![no_name](https://us1.discourse-cdn.com/elastic/original/3X/5/5/55306bf3beea05a70142487b912b63474af8b777.jpg) Is there some way to apply aggregation to all data, and not directly to each index inside index pattern? Or how to get data for all 100 items without skipping ? ---
**Author:** ![Waseem](https://avatars.discourse-cdn.com/v4/letter/w/9dc877/32.png) [@Waseem](https://discuss.elastic.co/u/Waseem)\ **Post date:** [March 21, 2018, 7:57am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/2 "2018-03-21T07:57:28Z")
+1 Same issue here. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 21, 2018, 10:34am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/3 "2018-03-21T10:34:56Z")
Try increase the `shard_size` parameter. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts) ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 22, 2018, 11:36am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/4 "2018-03-22T11:36:13Z")
Hi, thanks for reply. Can you show me where i can apply shard\_size in this aggregation structure ? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2018, 11:38am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/5 "2018-03-22T11:38:00Z")
> [@Grimmjow13r](#): > > an you show me where i can apply shard\_size in this aggregation structure ? Right alongside your `"size" : 100` parameter ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 22, 2018, 11:41am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/6 "2018-03-22T11:41:10Z")
Not helped ( still see the data gaps. here is the updated agg object ```auto "2": { "terms": { "field": "name", "size": 100, "shard_size": 500, "order": { "1": "desc" } }, "aggs": { "1": { "avg": { "field": "value" } }, "3": { "date_histogram": { "field": "@timestamp", "interval": "1d", "time_zone": "UTC", "min_doc_count": 1 }, "aggs": { "1": { "avg": { "field": "value" } } } } } } } ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2018, 11:55am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/7 "2018-03-22T11:55:00Z")
You'll likely need to increase it. There's a danger you can use a lot of memory and cause a circuit-breaker exception if you have a lot of unique terms - we'll then need to talk more about different strategies. ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 22, 2018, 12:03pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/8 "2018-03-22T12:03:15Z")
I tried "shard\_size": 100000000, nothing changed. the data gaps on the places. but if i'll set size to 200 all fine, no data gaps. But what i need is 100 items without data gaps, not more . ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2018, 12:28pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/9 "2018-03-22T12:28:31Z")
Strange. Roughly how many unique "name" values are there? (The `cardinality` aggregation can help tell you this) ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 22, 2018, 1:04pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/10 "2018-03-22T13:04:18Z")
104 unique name so with size 100 i see data gaps and 200 works correct. Seems that shard\_size not affecting on something ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2018, 1:15pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/11 "2018-03-22T13:15:41Z")
Are you checking the results for partial errors? When you query 5 shards successfully you should see 5/5 successes in the JSON response eg ``` "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 } ``` ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 22, 2018, 2:01pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/12 "2018-03-22T14:01:29Z")
yeah successful: 1 total:1 no failed or skipped ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2018, 2:14pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/13 "2018-03-22T14:14:22Z")
So you only have one index and one shard? That should make life even easier - there shouldn't be any of the usual concerns over `terms` accuracy and increasing `shard_size` etc. Two more questions - what elasticsearch version are you using and does it still fail to produce the correct results if you try remove the `min_doc_count:1` parameter on your date\_histogram agg? ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 22, 2018, 2:20pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/14 "2018-03-22T14:20:14Z")
es version is 5.2.2; Seems nothing changed when i removed min\_doc\_count. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2018, 2:40pm UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/16 "2018-03-22T14:40:31Z")
So assuming we have a passing test (size:200) and a failing test (size:100) let's try and simplify the aggregation to compare the results of these collections. Can you replace the date\_histogram aggregation with a simple `sum` aggregation on the `value` field. I'd like to know if the reported sums differ for the size:200 and size:100 queries. That should at least tell us if we're looking at the same set of docs/terms in the 2 queries. ---
**Author:** ![Grimmjow13r](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Grimmjow13r](https://discuss.elastic.co/u/Grimmjow13r)\ **Post date:** [March 29, 2018, 9:42am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/17 "2018-03-29T09:42:43Z")
Thanks! "shard\_size" helped in case of multiple indices. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 26, 2018, 9:42am UTC](https://discuss.elastic.co/t/aggregation-is-skipping-needed-data-and-not-give-expected-results/124837/18 "2018-04-26T09:42:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.