# How Facet information is aggregated in a cluster
**URL:** https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050
**Category:** Elasticsearch
**Created:** [December 12, 2012, 2:18am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050 "2012-12-12T02:18:01Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@revdev](https://discuss.elastic.co/u/revdev)
#### Post date: [December 12, 2012, 2:18am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/1 "2012-12-12T02:18:01Z")
Hi,
I wanted to know the behavior of the cluster when facet queries are run.
Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
with configuration of 4 shards and 1 replica.
If I issue a facet query spanning data which is present in multiple nodes,
which of the following happen?
Does the node receiving the facet request,
(1) query neighbors for data needed to perform the aggregation, then
aggregate on one node only (the one node will have a copy of all data from
other nodes needed for the aggregation temporarily)
or (2) query neighbors and ask for partial aggregation of data.
re-aggregate partial aggregated data. partial agg is trivial for sum, avg
I am asking because there were cases in my setup, I had around 16G RAM on
the cluster but the total data size was only 8G (including replica) and
even then some facet queries were causing OOM. I want to understand how
data is aggregated for for facets.
Secondly, I was wondering how can I ensure that I dont send too many
concurrent facet requests which might result in OOM. Can this even happen
or OOM can only happen if a single facet query is extremely large for data
to fit in memory?
--
---
### Author:  [@revdev](https://discuss.elastic.co/u/revdev)
#### Post date: [December 12, 2012, 4:53pm UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/2 "2012-12-12T16:53:12Z")
Any information on this would be really helpful to understand the behavior
we saw.
Thanks in advance!
Vinay
On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
> Hi,
> I wanted to know the behavior of the cluster when facet queries are run.
> Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
> with configuration of 4 shards and 1 replica.
> If I issue a facet query spanning data which is present in multiple nodes,
> which of the following happen?
>
> Does the node receiving the facet request,
> (1) query neighbors for data needed to perform the aggregation, then
> aggregate on one node only (the one node will have a copy of all data from
> other nodes needed for the aggregation temporarily)
> or (2) query neighbors and ask for partial aggregation of data.
> re-aggregate partial aggregated data. partial agg is trivial for sum, avg
>
> I am asking because there were cases in my setup, I had around 16G RAM on
> the cluster but the total data size was only 8G (including replica) and
> even then some facet queries were causing OOM. I want to understand how
> data is aggregated for for facets.
>
> Secondly, I was wondering how can I ensure that I dont send too many
> concurrent facet requests which might result in OOM. Can this even happen
> or OOM can only happen if a single facet query is extremely large for data
> to fit in memory?
--
---
### Author:  [@revdev](https://discuss.elastic.co/u/revdev)
#### Post date: [December 13, 2012, 6:17pm UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/3 "2012-12-13T18:17:09Z")
Anybody there? π
On Wednesday, December 12, 2012 8:53:12 AM UTC-8, revdev wrote:
> Any information on this would be really helpful to understand the behavior
> we saw.
> Thanks in advance!
> Vinay
>
> On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
>
> > Hi,
> > I wanted to know the behavior of the cluster when facet queries are run.
> > Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
> > with configuration of 4 shards and 1 replica.
> > If I issue a facet query spanning data which is present in multiple
> > nodes, which of the following happen?
> >
> > Does the node receiving the facet request,
> > (1) query neighbors for data needed to perform the aggregation, then
> > aggregate on one node only (the one node will have a copy of all data from
> > other nodes needed for the aggregation temporarily)
> > or (2) query neighbors and ask for partial aggregation of data.
> > re-aggregate partial aggregated data. partial agg is trivial for sum, avg
> >
> > I am asking because there were cases in my setup, I had around 16G RAM on
> > the cluster but the total data size was only 8G (including replica) and
> > even then some facet queries were causing OOM. I want to understand how
> > data is aggregated for for facets.
> >
> > Secondly, I was wondering how can I ensure that I dont send too many
> > concurrent facet requests which might result in OOM. Can this even happen
> > or OOM can only happen if a single facet query is extremely large for data
> > to fit in memory?
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [December 13, 2012, 6:23pm UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/4 "2012-12-13T18:23:45Z")
One item to look into are search types:
> **[Elasticsearch Platform β Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
The default is query\_then\_fetch. That said, I do not know what the
implications are for statistical facets. My assumption is that
with query\_then\_fetch, all calculations are done on the reducer.
Facet data is held in the field cache, which is inside the JVM. Profile
your app using tools such as BigDesk to see how large your field cache
grows. There has been some discussions about issues regarding memory and
facets.
>
>
> Hi,
>
> Our ElasticSearch instance contains circa 240 million messages. Each messagβ¦e can have one or more tags id associated with it. These ids are stored as an array of integers and we facet it using the standard terms facet.
>
> The facet cache size for this field is ~55GB which is highly surprising as only 5 million messages actually do have tags (tagging is manual). Also the total number of tag applications is only ~15 million.
>
> Yesterday our ES cluster died due to lack of memory.
>
> Researching it I have pin down the issue to the way the MultiValued\*Field caches work - For every segment it allocates memory space which is proportionate to the max number of values per docs \\\* maxDocs of that segment.
>
> In our case we had 3 messages with 100 tags which caused ElasticSearch to allocate 100\*24 million integers on 3 of the 10 shards we use (27.5 GB in total ). The rest of the shards each had at least one message with ~50 tags which is less dramatic but has a similar high consumption.
>
> I understand why the current MultiValueIntFieldData implementation is set as it is right now, but in our case it leads to extreme results.
>
> We currently worked around it by delete the tags from the top 200 messages which reduced memory considerably but this a short term solution.
>
> I have started working on a an alternative data structure which will solve things for us. I will submit a pull request as soon as it is ready.
>
> Cheers,
> Boaz
The current implementation is not ideal for fields with high cardinality.
Is your facet field multi-valued and can certains documents contain a
higher number of values compared to the others?
Cheers,
Ivan
On Thu, Dec 13, 2012 at 10:17 AM, revdev [clickingcam@gmail.com](mailto:clickingcam@gmail.com) wrote:
> Anybody there? π
>
> On Wednesday, December 12, 2012 8:53:12 AM UTC-8, revdev wrote:
>
> > Any information on this would be really helpful to understand the
> > behavior we saw.
> > Thanks in advance!
> > Vinay
> >
> > On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
> >
> > > Hi,
> > > I wanted to know the behavior of the cluster when facet queries are run.
> > > Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
> > > with configuration of 4 shards and 1 replica.
> > > If I issue a facet query spanning data which is present in multiple
> > > nodes, which of the following happen?
> > >
> > > Does the node receiving the facet request,
> > > (1) query neighbors for data needed to perform the aggregation, then
> > > aggregate on one node only (the one node will have a copy of all data from
> > > other nodes needed for the aggregation temporarily)
> > > or (2) query neighbors and ask for partial aggregation of data.
> > > re-aggregate partial aggregated data. partial agg is trivial for sum, avg
> > >
> > > I am asking because there were cases in my setup, I had around 16G RAM
> > > on the cluster but the total data size was only 8G (including replica) and
> > > even then some facet queries were causing OOM. I want to understand how
> > > data is aggregated for for facets.
> > >
> > > Secondly, I was wondering how can I ensure that I dont send too many
> > > concurrent facet requests which might result in OOM. Can this even happen
> > > or OOM can only happen if a single facet query is extremely large for data
> > > to fit in memory?
> > >
> > > --
--
---
### Author:  [@Vinay\_2](https://discuss.elastic.co/u/Vinay_2)
#### Post date: [December 14, 2012, 5:24pm UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/5 "2012-12-14T17:24:54Z")
Thanks Ivan. I see that there is lot of discussion going on in the group
about the very same issue of heap space optimization when it comes to
facets.
I am going to monitor field cache today while issuing heavy queries. Lot of
people recommend using "Soft Cache" but Shay has mentioned against it in
multiple threads. According to him soft cache will be invalidated often and
will have to rebuilt again and again which might be slow. I am just
wondering if it makes sense to use Soft Cache just to avoid cases of OOM.
If we have enough memory most of the time, soft caches should not be
invalidated unless we are really close to heap limit. This would make sure
in the few cases when ES get large queries, it doesn't crash because of
OOM.
About high cardinality. I am not sure how to figure that out since "high"
is a relative term. For example, the highest cardinality field in our
system is "dates" which stores second level precision. I can change it to
day level precision if required and if that significantly reduces field
cache size.
Do you know how I can test the max size that can be taken by field cache
for all of our data? Can I issue a single query which spans all documents
and calculate all possible facets for this purpose?
On Thu, Dec 13, 2012 at 10:23 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> One item to look into are search types:
>
> [Elasticsearch Platform β Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html)
>
> The default is query\_then\_fetch. That said, I do not know what the
> implications are for statistical facets. My assumption is that
> with query\_then\_fetch, all calculations are done on the reducer.
>
> Facet data is held in the field cache, which is inside the JVM. Profile
> your app using tools such as BigDesk to see how large your field cache
> grows. There has been some discussions about issues regarding memory and
> facets.
>
> [Unrealistic high memory consumption for faceting of infrequent array fields with many members Β· Issue #2468 Β· elastic/elasticsearch Β· GitHub](https://github.com/elasticsearch/elasticsearch/issues/2468)
>
> The current implementation is not ideal for fields with high cardinality.
> Is your facet field multi-valued and can certains documents contain a
> higher number of values compared to the others?
>
> Cheers,
>
> Ivan
>
> On Thu, Dec 13, 2012 at 10:17 AM, revdev [clickingcam@gmail.com](mailto:clickingcam@gmail.com) wrote:
>
> > Anybody there? π
> >
> > On Wednesday, December 12, 2012 8:53:12 AM UTC-8, revdev wrote:
> >
> > > Any information on this would be really helpful to understand the
> > > behavior we saw.
> > > Thanks in advance!
> > > Vinay
> > >
> > > On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
> > >
> > > > Hi,
> > > > I wanted to know the behavior of the cluster when facet queries are
> > > > run. Assuming, I have a cluster of 4 nodes and 4 indexes where each index
> > > > is with configuration of 4 shards and 1 replica.
> > > > If I issue a facet query spanning data which is present in multiple
> > > > nodes, which of the following happen?
> > > >
> > > > Does the node receiving the facet request,
> > > > (1) query neighbors for data needed to perform the aggregation, then
> > > > aggregate on one node only (the one node will have a copy of all data from
> > > > other nodes needed for the aggregation temporarily)
> > > > or (2) query neighbors and ask for partial aggregation of data.
> > > > re-aggregate partial aggregated data. partial agg is trivial for sum, avg
> > > >
> > > > I am asking because there were cases in my setup, I had around 16G RAM
> > > > on the cluster but the total data size was only 8G (including replica) and
> > > > even then some facet queries were causing OOM. I want to understand how
> > > > data is aggregated for for facets.
> > > >
> > > > Secondly, I was wondering how can I ensure that I dont send too many
> > > > concurrent facet requests which might result in OOM. Can this even happen
> > > > or OOM can only happen if a single facet query is extremely large for data
> > > > to fit in memory?
> > > >
> > > > --
>
> --
--
---
### Author:  [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [December 15, 2012, 8:08am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/6 "2012-12-15T08:08:53Z")
Hi Revdev,
You can run a match\_all query with a facet on every field you will need a facet on. This will load all the caches for these fields. You might want to want to run the facets one at time if your afraid that it might run OOM. After that you can get the cache size from the cluster nodes status API. As an alternative you can also use a pluging I wrote what I was in a similar situation : [https://github.com/bleskes/elasticfacets](https://github.com/bleskes/elasticfacets) . It has an end point to tell you the cache size on a field by field basis (it says its in development but it is fairly stable, we use it in production).
Cheers,
Boaz
--
---
### Author:  [@revdev](https://discuss.elastic.co/u/revdev)
#### Post date: [December 16, 2012, 1:43am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/7 "2012-12-16T01:43:09Z")
Thanks, thats a good idea. I'll check out your facet plugin too!
Vinay
On Saturday, December 15, 2012 12:08:53 AM UTC-8, Boaz Leskes wrote:
> Hi Revdev,
>
> You can run a match\_all query with a facet on every field you will need a
> facet on. This will load all the caches for these fields. You might want to
> want to run the facets one at time if your afraid that it might run OOM.
> After that you can get the cache size from the cluster nodes status API. As
> an alternative you can also use a pluging I wrote what I was in a similar
> situation : [GitHub - bleskes/elasticfacets: A set of facets and related tools for ElasticSearch](https://github.com/bleskes/elasticfacets) . It has an end
> point to tell you the cache size on a field by field basis (it says its in
> development but it is fairly stable, we use it in production).
>
> Cheers,
> Boaz
--
---
### Author:  [@phill](https://discuss.elastic.co/u/phill)
#### Post date: [January 3, 2013, 1:34am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/8 "2013-01-03T01:34:03Z")
There is no need to do _all_ calculations on the reducer. To go from
partial results from each node to total results for all nodes for
count, total, sum of squares, mean (average), minimum, maximum,
variance, and standard deviation
requires only a recalculation of few values based on each nodes partial
results.
overall std dev = sqrt(variance) = overall sum of squares / overall total
overall sum of squares = sum(each sum of squares)
etc.
I would hope like all other facets each node does all it can and the
"reduce" or "gather" phase has only a little work to do to calculation a
few values: O( #nodes ).
Pretty easy stuff for the "gather" phase.
-Paul
On 12/13/2012 10:23 AM, Ivan Brusic wrote:
> One item to look into are search types:
>
> [Elasticsearch Platform β Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html)
>
> The default is query\_then\_fetch. That said, I do not know what the
> implications are for statistical facets. My assumption is that
> with query\_then\_fetch, all calculations are done on the reducer.
--
---
### Author:  [@revdev](https://discuss.elastic.co/u/revdev)
#### Post date: [January 4, 2013, 1:50am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/9 "2013-01-04T01:50:48Z")
Thanks ! That explains it! π
On Wednesday, January 2, 2013 5:35:08 PM UTC-8, P Hill wrote:
> There is no need to do _all_ calculations on the reducer. To go from
> partial results from each node to total results for all nodes for
> count, total, sum of squares, mean (average), minimum, maximum,
> variance, and standard deviation
> requires only a recalculation of few values based on each nodes partial
> results.
> overall std dev = sqrt(variance) = overall sum of squares / overall total
> overall sum of squares = sum(each sum of squares)
> etc.
>
> I would hope like all other facets each node does all it can and the
> "reduce" or "gather" phase has only a little work to do to calculation a
> few values: O( #nodes ).
> Pretty easy stuff for the "gather" phase.
>
> -Paul
>
> On 12/13/2012 10:23 AM, Ivan Brusic wrote:
>
> > One item to look into are search types:
> >
> > [Elasticsearch Platform β Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html)
> >
> > The default is query\_then\_fetch. That said, I do not know what the
> > implications are for statistical facets. My assumption is that
> > with query\_then\_fetch, all calculations are done on the reducer.
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:57am UTC](https://discuss.elastic.co/t/how-facet-information-is-aggregated-in-a-cluster/10050/10 "2017-07-06T02:57:53Z")