# Questions about aggregation min\_doc\_count = 0

**URL:** <https://discuss.elastic.co/t/questions-about-aggregation-min-doc-count-0/16438>\
**Category:** Elasticsearch\
**Created:** [March 18, 2014, 7:39pm UTC](https://discuss.elastic.co/t/questions-about-aggregation-min-doc-count-0/16438 "2014-03-18T19:39:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![John\_Stanford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_stanford/32/1647_2.png) [@John\_Stanford](https://discuss.elastic.co/u/John_Stanford)\
**Post date:** [March 18, 2014, 7:39pm UTC](https://discuss.elastic.co/t/questions-about-aggregation-min-doc-count-0/16438/1 "2014-03-18T19:39:17Z")

</div>

Hi,

I'm trying to get a better understanding of aggregations, so here are a  
couple of questions that came up recently.

Question 1:

I have some time based data that I am using aggregations to chart. The  
data may be sparsely populated, so I've been setting min\_doc\_count to 0 so  
I get empty buckets back anyway. I've noticed that it will fill in empty  
buckets unless they are before or after the first record of the range.

For example, if I use a query similar to the one below, and there are no  
records after 3/15/14T16:15, the last aggregation record will be for  
3/15/14T16:15. On the other hand, if there is a gap in between the start  
time and 3/15/14T16:15, I will get a bucket with a 0 doc count (as  
expected).

POST \_all/summary\_phys/\_search

{  
"aggs": {  
"events\_by\_date": {  
"date\_histogram": {  
"field": "@timestamp",  
"interval": "300s",  
"min\_doc\_count": 0  
},  
"aggs": {  
"events\_by\_host": {  
"terms": {  
"field": "host.raw"  
},  
"aggs": {  
"avg\_used": {  
"avg": {  
"field": "used"  
}  
},  
"max\_used": {  
"max": {  
"field": "used"  
}  
}  
}  
}  
}  
}  
}  
}

Not getting the 0 doc count buckets back at the front and back of the range  
seems contrary to the documented purpose of min\_doc\_count. Am I doing  
something wrong?

Question 2:

If I add a min\_doc\_count = 0 to the inner aggregation, but limit the search  
to a specific doc type like:

```
                  doc type
                       v

```

POST \_all/summary\_phys/\_search  
{  
"aggs": {  
"events\_by\_date": {  
"date\_histogram": {  
"field": "@timestamp",  
"interval": "300s",  
"min\_doc\_count": 0  
},  
"aggs": {  
"events\_by\_host": {  
"terms": {  
"field": "host.raw",  
"min\_doc\_count": 0  
},  
"aggs": {  
"avg\_used": {  
"avg": {  
"field": "used"  
}  
},  
"max\_used": {  
"max": {  
"field": "used"  
}  
}  
}  
}  
}  
}  
}  
}

I get buckets with entries matching hosts that do not show up in this doc  
type. For example, I have only 3 values for host in this doc type  
[compute-4, compute-2, compute-3], but I will get buckets back with hosts  
from other doc types like:

"events\_by\_host": {  
"buckets": [  
{  
"key": "compute-4",  
"doc\_count": 11,  
"max\_used": {  
"value": 4608  
},  
"avg\_used": {  
"value": 3677.090909090909  
}  
},  
{  
"key": "compute-2",  
"doc\_count": 8,  
"max\_used": {  
"value": 4608  
},  
"avg\_used": {  
"value": 2304  
}  
},  
{  
"key": "compute-3",  
"doc\_count": 2,  
"max\_used": {  
"value": 4608  
},  
"avg\_used": {  
"value": 4608  
}  
},  
{  
"key": "10.10.11.22:49509",  
"doc\_count": 0,  
"max\_used": {  
"value": null  
},  
"avg\_used": {  
"value": null  
}  
},  
{  
"key": "controller",  
"doc\_count": 0,  
"max\_used": {  
"value": null  
},  
"avg\_used": {  
"value": null  
}  
},  
{  
"key": "object-1",  
"doc\_count": 0,  
"max\_used": {  
"value": null  
},  
"avg\_used": {  
"value": null  
}  
}  
]  
}

Is there a way to ensure that the inner aggregation also only buckets  
things matching the search doc type?

Thanks in advance...

John

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\
**Post date:** [March 18, 2014, 8:17pm UTC](https://discuss.elastic.co/t/questions-about-aggregation-min-doc-count-0/16438/2 "2014-03-18T20:17:10Z")

</div>

1. The histogram aggregation (and facet) work on indexed values not based  
on the current time or "now". So, if the last indexed document timestamp  
is 3/15/14T16:15 you will not get empty buckets between 3/15/14T16:15 and the  
current time. It would be interesting to be able to set the "to" and  
"from" on histogram based aggregations to allow for generating buckets on  
intervals between the defined range.

2. I believe this is the way the keys are pulled from the fielddata which  
is index level data. So if you are using the "all" index you are going to  
get data from all indices. Not sure if this is a bug or not. You can try  
applying a filter aggregation:

POST \_all/summary\_phys/\_search  
{  
"aggs": {  
"summary\_phys\_events": {  
"filter": {  
"type": {"value": "summary\_phys\_events"}  
},  
"aggs": {  
"events\_by\_date": {  
"date\_histogram": {  
"field": "@timestamp",  
"interval": "300s",  
"min\_doc\_count": 0  
},  
"aggs": {  
"events\_by\_host": {  
"terms": {  
"field": "host.raw",  
"min\_doc\_count": 0  
},  
"aggs": {  
"avg\_used": {  
"avg": {  
"field": "used"  
}  
},  
"max\_used": {  
"max": {  
"field": "used"  
}  
}  
}  
}  
}  
}  
}  
}  
}  
}

On Tue, Mar 18, 2014 at 12:39 PM, John Stanford [jxstanford@gmail.com](mailto:jxstanford@gmail.com)wrote:

> Hi,
> 
> I'm trying to get a better understanding of aggregations, so here are a  
> couple of questions that came up recently.
> 
> Question 1:
> 
> I have some time based data that I am using aggregations to chart. The  
> data may be sparsely populated, so I've been setting min\_doc\_count to 0 so  
> I get empty buckets back anyway. I've noticed that it will fill in empty  
> buckets unless they are before or after the first record of the range.
> 
> For example, if I use a query similar to the one below, and there are no  
> records after 3/15/14T16:15, the last aggregation record will be for  
> 3/15/14T16:15. On the other hand, if there is a gap in between the start  
> time and 3/15/14T16:15, I will get a bucket with a 0 doc count (as  
> expected).
> 
> POST \_all/summary\_phys/\_search
> 
> {  
> "aggs": {  
> "events\_by\_date": {  
> "date\_histogram": {  
> "field": "@timestamp",  
> "interval": "300s",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "events\_by\_host": {  
> "terms": {  
> "field": "host.raw"  
> },  
> "aggs": {  
> "avg\_used": {  
> "avg": {  
> "field": "used"  
> }  
> },  
> "max\_used": {  
> "max": {  
> "field": "used"  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }
> 
> Not getting the 0 doc count buckets back at the front and back of the  
> range seems contrary to the documented purpose of min\_doc\_count. Am I  
> doing something wrong?
> 
> Question 2:
> 
> If I add a min\_doc\_count = 0 to the inner aggregation, but limit the  
> search to a specific doc type like:
> 
> ```
> doc type
> v
> 
> ```
> 
> POST \_all/summary\_phys/\_search  
> {  
> "aggs": {  
> "events\_by\_date": {  
> "date\_histogram": {  
> "field": "@timestamp",  
> "interval": "300s",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "events\_by\_host": {  
> "terms": {  
> "field": "host.raw",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "avg\_used": {  
> "avg": {  
> "field": "used"  
> }  
> },  
> "max\_used": {  
> "max": {  
> "field": "used"  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }
> 
> I get buckets with entries matching hosts that do not show up in this doc  
> type. For example, I have only 3 values for host in this doc type  
> [compute-4, compute-2, compute-3], but I will get buckets back with hosts  
> from other doc types like:
> 
> "events\_by\_host": {  
> "buckets": [  
> {  
> "key": "compute-4",  
> "doc\_count": 11,  
> "max\_used": {  
> "value": 4608  
> },  
> "avg\_used": {  
> "value": 3677.090909090909  
> }  
> },  
> {  
> "key": "compute-2",  
> "doc\_count": 8,  
> "max\_used": {  
> "value": 4608  
> },  
> "avg\_used": {  
> "value": 2304  
> }  
> },  
> {  
> "key": "compute-3",  
> "doc\_count": 2,  
> "max\_used": {  
> "value": 4608  
> },  
> "avg\_used": {  
> "value": 4608  
> }  
> },  
> {  
> "key": "10.10.11.22:49509",  
> "doc\_count": 0,  
> "max\_used": {  
> "value": null  
> },  
> "avg\_used": {  
> "value": null  
> }  
> },  
> {  
> "key": "controller",  
> "doc\_count": 0,  
> "max\_used": {  
> "value": null  
> },  
> "avg\_used": {  
> "value": null  
> }  
> },  
> {  
> "key": "object-1",  
> "doc\_count": 0,  
> "max\_used": {  
> "value": null  
> },  
> "avg\_used": {  
> "value": null  
> }  
> }  
> ]  
> }
> 
> Is there a way to ensure that the inner aggregation also only buckets  
> things matching the search doc type?
> 
> Thanks in advance...
> 
> John
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoD1S47%2Bdu4hU8wAugzJW4LnWgP4A2XhjARLBnP2hvStJA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoD1S47%2Bdu4hU8wAugzJW4LnWgP4A2XhjARLBnP2hvStJA%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![John\_Stanford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_stanford/32/1647_2.png) [@John\_Stanford](https://discuss.elastic.co/u/John_Stanford)\
**Post date:** [March 19, 2014, 3:40am UTC](https://discuss.elastic.co/t/questions-about-aggregation-min-doc-count-0/16438/3 "2014-03-19T03:40:07Z")

</div>

Thanks Matt, I suspected as much on #1. I think it might save a little post-processing if it provided buckets for the specified range. The issue appears to be logged as [Create empty buckets in date\_/histogram aggregation at the edges, beyond the value space of the data · Issue #5224 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5224) and a pull request has been made. I tried the filter on #2, and it still picked up hosts that weren’t in that doc type, so I filed [filtering aggregations · Issue #5458 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5458).

Cheers,

John

[jxstanford@gmail.com](mailto:jxstanford@gmail.com)  
@jxstanford

On Mar 18, 2014, at 13:17:10, Matt Weber [matt.weber@gmail.com](mailto:matt.weber@gmail.com) wrote:

> 1. The histogram aggregation (and facet) work on indexed values not based on the current time or "now". So, if the last indexed document timestamp is 3/15/14T16:15 you will not get empty buckets between 3/15/14T16:15 and the current time. It would be interesting to be able to set the "to" and "from" on histogram based aggregations to allow for generating buckets on intervals between the defined range.
> 
> 2. I believe this is the way the keys are pulled from the fielddata which is index level data. So if you are using the "all" index you are going to get data from all indices. Not sure if this is a bug or not. You can try applying a filter aggregation:
> 
> POST \_all/summary\_phys/\_search  
> {  
> "aggs": {  
> "summary\_phys\_events": {  
> "filter": {  
> "type": {"value": "summary\_phys\_events"}  
> },  
> "aggs": {  
> "events\_by\_date": {  
> "date\_histogram": {  
> "field": "@timestamp",  
> "interval": "300s",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "events\_by\_host": {  
> "terms": {  
> "field": "host.raw",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "avg\_used": {  
> "avg": {  
> "field": "used"  
> }  
> },  
> "max\_used": {  
> "max": {  
> "field": "used"  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }
> 
> On Tue, Mar 18, 2014 at 12:39 PM, John Stanford [jxstanford@gmail.com](mailto:jxstanford@gmail.com) wrote:  
> Hi,
> 
> I'm trying to get a better understanding of aggregations, so here are a couple of questions that came up recently.
> 
> Question 1:
> 
> I have some time based data that I am using aggregations to chart. The data may be sparsely populated, so I've been setting min\_doc\_count to 0 so I get empty buckets back anyway. I've noticed that it will fill in empty buckets unless they are before or after the first record of the range.
> 
> For example, if I use a query similar to the one below, and there are no records after 3/15/14T16:15, the last aggregation record will be for 3/15/14T16:15. On the other hand, if there is a gap in between the start time and 3/15/14T16:15, I will get a bucket with a 0 doc count (as expected).
> 
> POST \_all/summary\_phys/\_search
> 
> {  
> "aggs": {  
> "events\_by\_date": {  
> "date\_histogram": {  
> "field": "@timestamp",  
> "interval": "300s",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "events\_by\_host": {  
> "terms": {  
> "field": "host.raw"  
> },  
> "aggs": {  
> "avg\_used": {  
> "avg": {  
> "field": "used"  
> }  
> },  
> "max\_used": {  
> "max": {  
> "field": "used"  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }
> 
> Not getting the 0 doc count buckets back at the front and back of the range seems contrary to the documented purpose of min\_doc\_count. Am I doing something wrong?
> 
> Question 2:
> 
> If I add a min\_doc\_count = 0 to the inner aggregation, but limit the search to a specific doc type like:
> 
> ```
> doc type
> v
> 
> ```
> 
> POST \_all/summary\_phys/\_search  
> {  
> "aggs": {  
> "events\_by\_date": {  
> "date\_histogram": {  
> "field": "@timestamp",  
> "interval": "300s",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "events\_by\_host": {  
> "terms": {  
> "field": "host.raw",  
> "min\_doc\_count": 0  
> },  
> "aggs": {  
> "avg\_used": {  
> "avg": {  
> "field": "used"  
> }  
> },  
> "max\_used": {  
> "max": {  
> "field": "used"  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }
> 
> I get buckets with entries matching hosts that do not show up in this doc type. For example, I have only 3 values for host in this doc type [compute-4, compute-2, compute-3], but I will get buckets back with hosts from other doc types like:
> 
> "events\_by\_host": {  
> "buckets": [  
> {  
> "key": "compute-4",  
> "doc\_count": 11,  
> "max\_used": {  
> "value": 4608  
> },  
> "avg\_used": {  
> "value": 3677.090909090909  
> }  
> },  
> {  
> "key": "compute-2",  
> "doc\_count": 8,  
> "max\_used": {  
> "value": 4608  
> },  
> "avg\_used": {  
> "value": 2304  
> }  
> },  
> {  
> "key": "compute-3",  
> "doc\_count": 2,  
> "max\_used": {  
> "value": 4608  
> },  
> "avg\_used": {  
> "value": 4608  
> }  
> },  
> {  
> "key": "10.10.11.22:49509",  
> "doc\_count": 0,  
> "max\_used": {  
> "value": null  
> },  
> "avg\_used": {  
> "value": null  
> }  
> },  
> {  
> "key": "controller",  
> "doc\_count": 0,  
> "max\_used": {  
> "value": null  
> },  
> "avg\_used": {  
> "value": null  
> }  
> },  
> {  
> "key": "object-1",  
> "doc\_count": 0,  
> "max\_used": {  
> "value": null  
> },  
> "avg\_used": {  
> "value": null  
> }  
> }  
> ]  
> }
> 
> Is there a way to ensure that the inner aggregation also only buckets things matching the search doc type?
> 
> Thanks in advance...
> 
> John
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/856133dc-c4ae-4cfc-adab-39453671d76d%40googlegroups.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/kz0eFP7nZMU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kz0eFP7nZMU/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoD1S47%2Bdu4hU8wAugzJW4LnWgP4A2XhjARLBnP2hvStJA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoD1S47%2Bdu4hU8wAugzJW4LnWgP4A2XhjARLBnP2hvStJA%40mail.gmail.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:42am UTC](https://discuss.elastic.co/t/questions-about-aggregation-min-doc-count-0/16438/4 "2017-07-06T01:42:20Z")

</div>