# Accuracy issue of aggregation results
**URL:** https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823
**Category:** Elasticsearch
**Created:** [September 16, 2014, 7:36pm UTC](https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823 "2014-09-16T19:36:16Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Yifan\_Wang](https://discuss.elastic.co/u/Yifan_Wang)
#### Post date: [September 16, 2014, 7:36pm UTC](https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823/1 "2014-09-16T19:36:16Z")
It seems to be a common problem that the top N results returned from an
aggregation query is inaccurate due to uneven distribution of matching
documents on different shards, because ES will collect top N buckets from
each shard no matter actually how many hits are on each shard. It is very
often we collect buckets that should have not been collected on some
shards, but we missed buckets that should have collected on some others.
Is there a way we can collect buckets based on a dynamic "weight", for
example "total hits", on that shard?
Thanks in advance.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@mattweber](https://discuss.elastic.co/u/mattweber)
#### Post date: [September 16, 2014, 8:20pm UTC](https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823/2 "2014-09-16T20:20:42Z")
Hi Yifan,
Nothing dynamic, but you can increase the number of terms collected on each
shard to increase the accuracy [1]. Might also want to play with the
shard\_min\_doc\_count value if you know certain shards have a low hit count
and are throwing off the aggregations [2].
[1]
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
[2]
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
Thanks,
Matt Weber
On Tue, Sep 16, 2014 at 12:36 PM, Yifan Wang [yifan.wang.usa@gmail.com](mailto:yifan.wang.usa@gmail.com)
wrote:
> It seems to be a common problem that the top N results returned from an
> aggregation query is inaccurate due to uneven distribution of matching
> documents on different shards, because ES will collect top N buckets from
> each shard no matter actually how many hits are on each shard. It is very
> often we collect buckets that should have not been collected on some
> shards, but we missed buckets that should have collected on some others.
>
> Is there a way we can collect buckets based on a dynamic "weight", for
> example "total hits", on that shard?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoCWieyr%3DW%2B\_T0wxPr9L6\_USLMKNQuMTNx0MOBQAaZ\_VQA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoCWieyr%3DW%2B_T0wxPr9L6_USLMKNQuMTNx0MOBQAaZ_VQA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Yifan\_Wang](https://discuss.elastic.co/u/Yifan_Wang)
#### Post date: [September 16, 2014, 9:09pm UTC](https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823/3 "2014-09-16T21:09:51Z")
Hi Matt,
Thanks for your quick response. However neither worked for us. In our case,
we set shard\_size to 50K (option1 ), it is still missing documents. The
cluster became unstable if we try to further increase it. We cannot use
shard\_min\_doc\_count\_value, because even it is one hit, its value used for
bucket ordering can still be large enough to be collected. What we really
need is "weighted" collect. As a workaround we have to do multiple trips.
"Weighted collect" may have some performance penalty, but it would be
better option than multiple trips or setting large shard\_size. I am
wondering if ES plugin can achieve this goal.
Thanks.
On Tuesday, September 16, 2014 4:20:55 PM UTC-4, Matt Weber wrote:
> Hi Yifan,
>
> Nothing dynamic, but you can increase the number of terms collected on
> each shard to increase the accuracy [1]. Might also want to play with the
> shard\_min\_doc\_count value if you know certain shards have a low hit count
> and are throwing off the aggregations [2].
>
> [1]
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_shard_size)
> [2]
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_minimum_document_count)
>
> Thanks,
> Matt Weber
>
> On Tue, Sep 16, 2014 at 12:36 PM, Yifan Wang \<[yifan.w...@gmail.com](mailto:yifan.w...@gmail.com)
> \\> wrote:
>
> > It seems to be a common problem that the top N results returned from an
> > aggregation query is inaccurate due to uneven distribution of matching
> > documents on different shards, because ES will collect top N buckets from
> > each shard no matter actually how many hits are on each shard. It is very
> > often we collect buckets that should have not been collected on some
> > shards, but we missed buckets that should have collected on some others.
> >
> > Is there a way we can collect buckets based on a dynamic "weight", for
> > example "total hits", on that shard?
> >
> > Thanks in advance.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ff23136d-eea3-4863-bec1-3caa8edf4777%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ff23136d-eea3-4863-bec1-3caa8edf4777%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Yifan\_Wang](https://discuss.elastic.co/u/Yifan_Wang)
#### Post date: [December 17, 2014, 10:24pm UTC](https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823/4 "2014-12-17T22:24:57Z")
Just in case anyone is interested, "weighted collect" (collect more on
shards of more documents) actually does not necessarily improve the
accuracy if the documents are distributed by default hash algorithm. There
is no such correlations.
On Tuesday, September 16, 2014 5:09:51 PM UTC-4, Yifan Wang wrote:
> Hi Matt,
>
> Thanks for your quick response. However neither worked for us. In our
> case, we set shard\_size to 50K (option1 ), it is still missing documents.
> The cluster became unstable if we try to further increase it. We cannot use
> shard\_min\_doc\_count\_value, because even it is one hit, its value used for
> bucket ordering can still be large enough to be collected. What we really
> need is "weighted" collect. As a workaround we have to do multiple trips.
> "Weighted collect" may have some performance penalty, but it would be
> better option than multiple trips or setting large shard\_size. I am
> wondering if ES plugin can achieve this goal.
>
> Thanks.
>
> On Tuesday, September 16, 2014 4:20:55 PM UTC-4, Matt Weber wrote:
>
> > Hi Yifan,
> >
> > Nothing dynamic, but you can increase the number of terms collected on
> > each shard to increase the accuracy [1]. Might also want to play with the
> > shard\_min\_doc\_count value if you know certain shards have a low hit count
> > and are throwing off the aggregations [2].
> >
> > [1]
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_shard_size)
> > [2]
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_minimum_document_count)
> >
> > Thanks,
> > Matt Weber
> >
> > On Tue, Sep 16, 2014 at 12:36 PM, Yifan Wang [yifan.w...@gmail.com](mailto:yifan.w...@gmail.com)
> > wrote:
> >
> > > It seems to be a common problem that the top N results returned from an
> > > aggregation query is inaccurate due to uneven distribution of matching
> > > documents on different shards, because ES will collect top N buckets from
> > > each shard no matter actually how many hits are on each shard. It is very
> > > often we collect buckets that should have not been collected on some
> > > shards, but we missed buckets that should have collected on some others.
> > >
> > > Is there a way we can collect buckets based on a dynamic "weight", for
> > > example "total hits", on that shard?
> > >
> > > Thanks in advance.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e78571f9-d3e3-4d7c-a60e-d1a2052db397%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/393f139e-a8df-46e9-bea1-374460958a36%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/393f139e-a8df-46e9-bea1-374460958a36%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:43am UTC](https://discuss.elastic.co/t/accuracy-issue-of-aggregation-results/19823/5 "2017-07-06T00:43:04Z")