# Does date histogram load entire field into memory, even with range filter?

**URL:** <https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118>\
**Category:** Elasticsearch\
**Created:** [June 15, 2012, 11:06am UTC](https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118 "2012-06-15T11:06:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrew\_Clegg](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@Andrew\_Clegg](https://discuss.elastic.co/u/Andrew_Clegg)\
**Post date:** [June 15, 2012, 11:06am UTC](https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118/1 "2012-06-15T11:06:10Z")

</div>

Hi,

I'm trying to run a date facet over a subset of a long time series (very  
many values), and it keeps OOMing. But when I remove the facet clause from  
the query, I get an overall result instantly.

This suggests to me that even with the filter in place, ES is trying to  
load all the distinct values of the field. Is that correct? If so, is there  
any way round it?

The query looks like this:

{  
"query": {  
"filtered" : {  
"query" : {  
"range" : {  
"pubDate" : {  
"from" : "2010-10-01",  
"to" : "2011-01-01"  
}  
}  
},  
"filter" : {  
"and" : {  
"filters": [  
{  
"exists" : { "field" : "foo" }  
},  
{  
"term" : { "bar" : "somestring" }  
},  
{  
"prefix" : { "baz" : "a" }  
}  
]  
}  
}  
}  
},  
"facets" : {  
"published" : {  
"date\_histogram" : {  
"field" : "pubDate",  
"interval" : "month"  
}  
}  
}  
}

Like that, I get:

[2012-06-15 11:54:48,335][WARN][index.cache.field.data.soft] [Centurion]  
[search\_criteria] loading field [event.pubDate] caused out of memory failure

But when I take out the facet, no problems.

This is with search\_type=count by the way, as I don't care about the actual  
hits.

Thanks,

Andrew.

---

<div class="post-metadata">

**Author:** ![Andrew\_Clegg](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@Andrew\_Clegg](https://discuss.elastic.co/u/Andrew_Clegg)\
**Post date:** [June 15, 2012, 11:18am UTC](https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118/2 "2012-06-15T11:18:47Z")

</div>

Sorry, I realise I got the terminology wrong here, what I meant was a range  
_query_ not a range _filter_.

On Friday, 15 June 2012 12:06:10 UTC+1, Andrew Clegg wrote:

> Hi,
> 
> I'm trying to run a date facet over a subset of a long time series (very  
> many values), and it keeps OOMing. But when I remove the facet clause from  
> the query, I get an overall result instantly.
> 
> This suggests to me that even with the filter in place, ES is trying to  
> load all the distinct values of the field. Is that correct? If so, is there  
> any way round it?
> 
> The query looks like this:
> 
> {  
> "query": {  
> "filtered" : {  
> "query" : {  
> "range" : {  
> "pubDate" : {  
> "from" : "2010-10-01",  
> "to" : "2011-01-01"  
> }  
> }  
> },  
> "filter" : {  
> "and" : {  
> "filters": [  
> {  
> "exists" : { "field" : "foo" }  
> },  
> {  
> "term" : { "bar" : "somestring" }  
> },  
> {  
> "prefix" : { "baz" : "a" }  
> }  
> ]  
> }  
> }  
> }  
> },  
> "facets" : {  
> "published" : {  
> "date\_histogram" : {  
> "field" : "pubDate",  
> "interval" : "month"  
> }  
> }  
> }  
> }
> 
> Like that, I get:
> 
> [2012-06-15 11:54:48,335][WARN][index.cache.field.data.soft] [Centurion]  
> [search\_criteria] loading field [event.pubDate] caused out of memory failure
> 
> But when I take out the facet, no problems.
> 
> This is with search\_type=count by the way, as I don't care about the  
> actual hits.
> 
> Thanks,
> 
> Andrew.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 15, 2012, 11:24am UTC](https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118/3 "2012-06-15T11:24:07Z")

</div>

Hi Andrew,

You have to filter the facet with the same filters you are already using in your  
query.  
So put your range filter as a Facet Filter should help.

Facet Filter

All facets can be configured with an additional filter (explained in the Query  
DSL [http://www.elasticsearch.org/guide/reference/query-dsl](http://www.elasticsearch.org/guide/reference/query-dsl) section), which  
will reduce the documents they use for computing results. An example with a term  
filter:

{  
"facets" : {  
"" : {  
"" : {  
...  
},  
"facet\_filter" : {  
"term" : { "user" : "kimchy"}  
}  
}  
}  
}

Note that this is different from a facet of the filter  
[http://www.elasticsearch.org/guide/reference/api/search/facets/filter-facet.html](http://www.elasticsearch.org/guide/reference/api/search/facets/filter-facet.html)  
type.

See also if scope could help :

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

[http://www.elasticsearch.org/guide/reference/api/search/facets/index.html](http://www.elasticsearch.org/guide/reference/api/search/facets/index.html)

Scope

As we have already mentioned, facet computation is restricted to the scope of  
the current query, called main, by default. Facets can be computed within the  
global scope as well, in which case it will return values computed acrosss all  
documents in the index:

{  
"facets" : {  
"" : {  
"" : { ... },  
"global" : true  
}  
}  
}

There’s one important distinction to keep in mind. While search queries restrict  
both the returned documents and facet counts, search filters restrict only  
returned documents — but notfacet counts.

If you need to restrict both the documents and facets, and you’re not willing or  
able to use a query, you may use a facet filter.

HTH  
David.

Le 15 juin 2012 à 13:06, Andrew Clegg [andrew.clegg@gmail.com](mailto:andrew.clegg@gmail.com) a écrit :

> Hi,
> 
> I'm trying to run a date facet over a subset of a long time series (very many  
> values), and it keeps OOMing. But when I remove the facet clause from the  
> query, I get an overall result instantly.
> 
> This suggests to me that even with the filter in place, ES is trying to load  
> all the distinct values of the field. Is that correct? If so, is there any way  
> round it?
> 
> The query looks like this:
> 
> {  
> "query": {  
> "filtered" : {  
> "query" : {  
> "range" : {  
> "pubDate" : {  
> "from" : "2010-10-01",  
> "to" : "2011-01-01"  
> }  
> }  
> },  
> "filter" : {  
> "and" : {  
> "filters": [  
> {  
> "exists" : { "field" : "foo" }  
> },  
> {  
> "term" : { "bar" : "somestring" }  
> },  
> {  
> "prefix" : { "baz" : "a" }  
> }  
> ]  
> }  
> }  
> }  
> },  
> "facets" : {  
> "published" : {  
> "date\_histogram" : {  
> "field" : "pubDate",  
> "interval" : "month"  
> }  
> }  
> }  
> }
> 
> Like that, I get:
> 
> [2012-06-15 11:54:48,335][WARN][index.cache.field.data.soft] [Centurion]  
> [search\_criteria] loading field [event.pubDate] caused out of memory failure
> 
> But when I take out the facet, no problems.
> 
> This is with search\_type=count by the way, as I don't care about the actual  
> hits.
> 
> Thanks,
> 
> Andrew.

--  
David Pilato  
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)  
Twitter : @dadoonet

---

<div class="post-metadata">

**Author:** ![Andrew\_Clegg](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@Andrew\_Clegg](https://discuss.elastic.co/u/Andrew_Clegg)\
**Post date:** [June 15, 2012, 11:40am UTC](https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118/4 "2012-06-15T11:40:04Z")

</div>

Hi David, I'm not sure that's helped, sadly.

I've tried making the query as restrictive as possible, by putting the same  
filters in both the main query and the facet:

{  
"query": {  
"filtered" : {  
"query" : {  
"match\_all" : { }  
},  
"filter" : {  
"and" : {  
"filters": [  
{ "range" : {  
"pubDate" : {  
"from" : "2010-12-31",  
"to" : "2011-01-01"  
}  
} },  
{  
"exists" : { "field" : "foo" }  
},  
{  
"term" : { "bar" : "XXX" }  
},  
{  
"prefix" : { "baz" : "a" }  
}  
]  
}  
}  
}  
},  
"facets" : {  
"published" : {  
"date\_histogram" : {  
"field" : "pubDate",  
"interval" : "month"  
},  
"facet\_filter" : {  
"and" : {  
"filters": [  
{ "range" : {  
"pubDate" : {  
"from" : "2010-12-31",  
"to" : "2011-01-01"  
}  
} },  
{  
"exists" : { "field" : "foo" }  
},  
{  
"term" : { "bar" : "XXX" }  
},  
{  
"prefix" : { "baz" : "a" }  
}  
]  
}  
}  
}  
}  
}

but even this with a _single_ day covered bombs out in a JVM with 1G heap.

And there are only 63388 documents in that day so there's no reason it  
should. (I know this because a count query without a facet on that date  
range is instant...)

On Friday, 15 June 2012 12:24:07 UTC+1, David Pilato wrote:

> Hi Andrew,
> 
> You have to filter the facet with the same filters you are already using  
> in your query.
> 
> So put your range filter as a Facet Filter should help.  
> Facet Filter
> 
> All facets can be configured with an additional filter (explained in the  
> Query DSL [http://www.elasticsearch.org/guide/reference/query-dsl](http://www.elasticsearch.org/guide/reference/query-dsl) section),  
> which _will_ reduce the documents they use for computing results. An  
> example with a _term_ filter:
> 
> {  
> "facets" : {  
> "" : {  
> "" : {  
> ...  
> },  
> "facet\_filter" : {  
> "term" : { "user" : "kimchy"}  
> }  
> }  
> }  
> }
> 
> Note that this is different from a facet of the filter[http://www.elasticsearch.org/guide/reference/api/search/facets/filter-facet.html](http://www.elasticsearch.org/guide/reference/api/search/facets/filter-facet.html)  
> type.
> 
> See also if scope could help :  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/facets/index.html)  
> Scope
> 
> As we have already mentioned, facet computation is restricted to the scope  
> of the current query, called main, by default. Facets can be computed  
> within the global scope as well, in which case it will return values  
> computed acrosss all documents in the index:
> 
> {  
> "facets" : {  
> "" : {  
> "" : { ... },  
> "global" : true  
> }  
> }  
> }
> 
> There’s one _important distinction_ to keep in mind. While search \*queries
> 
> - restrict both the returned documents and facet counts, search _filters_ restrict  
> only returned documents — but _not_facet counts.
> 
> If you need to restrict both the documents and facets, and you’re not  
> willing or able to use a query, you may use a _facet filter_.
> 
> HTH
> 
> David.
> 
> Le 15 juin 2012 à 13:06, Andrew Clegg [andrew.clegg@gmail.com](mailto:andrew.clegg@gmail.com) a écrit :
> 
> Hi,
> 
> I'm trying to run a date facet over a subset of a long time series (very  
> many values), and it keeps OOMing. But when I remove the facet clause from  
> the query, I get an overall result instantly.
> 
> This suggests to me that even with the filter in place, ES is trying to  
> load all the distinct values of the field. Is that correct? If so, is there  
> any way round it?
> 
> The query looks like this:
> 
> {  
> "query": {  
> "filtered" : {  
> "query" : {  
> "range" : {  
> "pubDate" : {  
> "from" : "2010-10-01",  
> "to" : "2011-01-01"  
> }  
> }  
> },  
> "filter" : {  
> "and" : {  
> "filters": [  
> {  
> "exists" : { "field" : "foo" }  
> },  
> {  
> "term" : { "bar" : "somestring" }  
> },  
> {  
> "prefix" : { "baz" : "a" }  
> }  
> ]  
> }  
> }  
> }  
> },  
> "facets" : {  
> "published" : {  
> "date\_histogram" : {  
> "field" : "pubDate",  
> "interval" : "month"  
> }  
> }  
> }  
> }
> 
> Like that, I get:
> 
> [2012-06-15 11:54:48,335][WARN][index.cache.field.data.soft] [Centurion]  
> [search\_criteria] loading field [event.pubDate] caused out of memory  
> failure
> 
> But when I take out the facet, no problems.
> 
> This is with search\_type=count by the way, as I don't care about the  
> actual hits.
> 
> Thanks,
> 
> Andrew.
> 
> --  
> David Pilato  
> [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)  
> Twitter : @dadoonet

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:23am UTC](https://discuss.elastic.co/t/does-date-histogram-load-entire-field-into-memory-even-with-range-filter/8118/5 "2017-07-06T03:23:54Z")

</div>