Hi,
I wanted to know the behavior of the cluster when facet queries are run.
Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
with configuration of 4 shards and 1 replica.
If I issue a facet query spanning data which is present in multiple nodes,
which of the following happen?
Does the node receiving the facet request,
(1) query neighbors for data needed to perform the aggregation, then
aggregate on one node only (the one node will have a copy of all data from
other nodes needed for the aggregation temporarily)
or (2) query neighbors and ask for partial aggregation of data.
re-aggregate partial aggregated data. partial agg is trivial for sum, avg
I am asking because there were cases in my setup, I had around 16G RAM on
the cluster but the total data size was only 8G (including replica) and
even then some facet queries were causing OOM. I want to understand how
data is aggregated for for facets.
Secondly, I was wondering how can I ensure that I dont send too many
concurrent facet requests which might result in OOM. Can this even happen
or OOM can only happen if a single facet query is extremely large for data
to fit in memory?
Any information on this would be really helpful to understand the behavior
we saw.
Thanks in advance!
Vinay
On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
Hi,
I wanted to know the behavior of the cluster when facet queries are run.
Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
with configuration of 4 shards and 1 replica.
If I issue a facet query spanning data which is present in multiple nodes,
which of the following happen?
Does the node receiving the facet request,
(1) query neighbors for data needed to perform the aggregation, then
aggregate on one node only (the one node will have a copy of all data from
other nodes needed for the aggregation temporarily)
or (2) query neighbors and ask for partial aggregation of data.
re-aggregate partial aggregated data. partial agg is trivial for sum, avg
I am asking because there were cases in my setup, I had around 16G RAM on
the cluster but the total data size was only 8G (including replica) and
even then some facet queries were causing OOM. I want to understand how
data is aggregated for for facets.
Secondly, I was wondering how can I ensure that I dont send too many
concurrent facet requests which might result in OOM. Can this even happen
or OOM can only happen if a single facet query is extremely large for data
to fit in memory?
On Wednesday, December 12, 2012 8:53:12 AM UTC-8, revdev wrote:
Any information on this would be really helpful to understand the behavior
we saw.
Thanks in advance!
Vinay
On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
Hi,
I wanted to know the behavior of the cluster when facet queries are run.
Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
with configuration of 4 shards and 1 replica.
If I issue a facet query spanning data which is present in multiple
nodes, which of the following happen?
Does the node receiving the facet request,
(1) query neighbors for data needed to perform the aggregation, then
aggregate on one node only (the one node will have a copy of all data from
other nodes needed for the aggregation temporarily)
or (2) query neighbors and ask for partial aggregation of data.
re-aggregate partial aggregated data. partial agg is trivial for sum, avg
I am asking because there were cases in my setup, I had around 16G RAM on
the cluster but the total data size was only 8G (including replica) and
even then some facet queries were causing OOM. I want to understand how
data is aggregated for for facets.
Secondly, I was wondering how can I ensure that I dont send too many
concurrent facet requests which might result in OOM. Can this even happen
or OOM can only happen if a single facet query is extremely large for data
to fit in memory?
The default is query_then_fetch. That said, I do not know what the
implications are for statistical facets. My assumption is that
with query_then_fetch, all calculations are done on the reducer.
Facet data is held in the field cache, which is inside the JVM. Profile
your app using tools such as BigDesk to see how large your field cache
grows. There has been some discussions about issues regarding memory and
facets.
The current implementation is not ideal for fields with high cardinality.
Is your facet field multi-valued and can certains documents contain a
higher number of values compared to the others?
On Wednesday, December 12, 2012 8:53:12 AM UTC-8, revdev wrote:
Any information on this would be really helpful to understand the
behavior we saw.
Thanks in advance!
Vinay
On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
Hi,
I wanted to know the behavior of the cluster when facet queries are run.
Assuming, I have a cluster of 4 nodes and 4 indexes where each index is
with configuration of 4 shards and 1 replica.
If I issue a facet query spanning data which is present in multiple
nodes, which of the following happen?
Does the node receiving the facet request,
(1) query neighbors for data needed to perform the aggregation, then
aggregate on one node only (the one node will have a copy of all data from
other nodes needed for the aggregation temporarily)
or (2) query neighbors and ask for partial aggregation of data.
re-aggregate partial aggregated data. partial agg is trivial for sum, avg
I am asking because there were cases in my setup, I had around 16G RAM
on the cluster but the total data size was only 8G (including replica) and
even then some facet queries were causing OOM. I want to understand how
data is aggregated for for facets.
Secondly, I was wondering how can I ensure that I dont send too many
concurrent facet requests which might result in OOM. Can this even happen
or OOM can only happen if a single facet query is extremely large for data
to fit in memory?
Thanks Ivan. I see that there is lot of discussion going on in the group
about the very same issue of heap space optimization when it comes to
facets.
I am going to monitor field cache today while issuing heavy queries. Lot of
people recommend using "Soft Cache" but Shay has mentioned against it in
multiple threads. According to him soft cache will be invalidated often and
will have to rebuilt again and again which might be slow. I am just
wondering if it makes sense to use Soft Cache just to avoid cases of OOM.
If we have enough memory most of the time, soft caches should not be
invalidated unless we are really close to heap limit. This would make sure
in the few cases when ES get large queries, it doesn't crash because of
OOM.
About high cardinality. I am not sure how to figure that out since "high"
is a relative term. For example, the highest cardinality field in our
system is "dates" which stores second level precision. I can change it to
day level precision if required and if that significantly reduces field
cache size.
Do you know how I can test the max size that can be taken by field cache
for all of our data? Can I issue a single query which spans all documents
and calculate all possible facets for this purpose?
On Thu, Dec 13, 2012 at 10:23 AM, Ivan Brusic ivan@brusic.com wrote:
The default is query_then_fetch. That said, I do not know what the
implications are for statistical facets. My assumption is that
with query_then_fetch, all calculations are done on the reducer.
Facet data is held in the field cache, which is inside the JVM. Profile
your app using tools such as BigDesk to see how large your field cache
grows. There has been some discussions about issues regarding memory and
facets.
The current implementation is not ideal for fields with high cardinality.
Is your facet field multi-valued and can certains documents contain a
higher number of values compared to the others?
On Wednesday, December 12, 2012 8:53:12 AM UTC-8, revdev wrote:
Any information on this would be really helpful to understand the
behavior we saw.
Thanks in advance!
Vinay
On Tuesday, December 11, 2012 6:18:01 PM UTC-8, revdev wrote:
Hi,
I wanted to know the behavior of the cluster when facet queries are
run. Assuming, I have a cluster of 4 nodes and 4 indexes where each index
is with configuration of 4 shards and 1 replica.
If I issue a facet query spanning data which is present in multiple
nodes, which of the following happen?
Does the node receiving the facet request,
(1) query neighbors for data needed to perform the aggregation, then
aggregate on one node only (the one node will have a copy of all data from
other nodes needed for the aggregation temporarily)
or (2) query neighbors and ask for partial aggregation of data.
re-aggregate partial aggregated data. partial agg is trivial for sum, avg
I am asking because there were cases in my setup, I had around 16G RAM
on the cluster but the total data size was only 8G (including replica) and
even then some facet queries were causing OOM. I want to understand how
data is aggregated for for facets.
Secondly, I was wondering how can I ensure that I dont send too many
concurrent facet requests which might result in OOM. Can this even happen
or OOM can only happen if a single facet query is extremely large for data
to fit in memory?
You can run a match_all query with a facet on every field you will need a facet on. This will load all the caches for these fields. You might want to want to run the facets one at time if your afraid that it might run OOM. After that you can get the cache size from the cluster nodes status API. As an alternative you can also use a pluging I wrote what I was in a similar situation : https://github.com/bleskes/elasticfacets . It has an end point to tell you the cache size on a field by field basis (it says its in development but it is fairly stable, we use it in production).
Thanks, thats a good idea. I'll check out your facet plugin too!
Vinay
On Saturday, December 15, 2012 12:08:53 AM UTC-8, Boaz Leskes wrote:
Hi Revdev,
You can run a match_all query with a facet on every field you will need a
facet on. This will load all the caches for these fields. You might want to
want to run the facets one at time if your afraid that it might run OOM.
After that you can get the cache size from the cluster nodes status API. As
an alternative you can also use a pluging I wrote what I was in a similar
situation : GitHub - bleskes/elasticfacets: A set of facets and related tools for ElasticSearch . It has an end
point to tell you the cache size on a field by field basis (it says its in
development but it is fairly stable, we use it in production).
There is no need to do all calculations on the reducer. To go from
partial results from each node to total results for all nodes for
count, total, sum of squares, mean (average), minimum, maximum,
variance, and standard deviation
requires only a recalculation of few values based on each nodes partial
results.
overall std dev = sqrt(variance) = overall sum of squares / overall total
overall sum of squares = sum(each sum of squares)
etc.
I would hope like all other facets each node does all it can and the
"reduce" or "gather" phase has only a little work to do to calculation a
few values: O( #nodes ).
Pretty easy stuff for the "gather" phase.
The default is query_then_fetch. That said, I do not know what the
implications are for statistical facets. My assumption is that
with query_then_fetch, all calculations are done on the reducer.
On Wednesday, January 2, 2013 5:35:08 PM UTC-8, P Hill wrote:
There is no need to do all calculations on the reducer. To go from
partial results from each node to total results for all nodes for
count, total, sum of squares, mean (average), minimum, maximum,
variance, and standard deviation
requires only a recalculation of few values based on each nodes partial
results.
overall std dev = sqrt(variance) = overall sum of squares / overall total
overall sum of squares = sum(each sum of squares)
etc.
I would hope like all other facets each node does all it can and the
"reduce" or "gather" phase has only a little work to do to calculation a
few values: O( #nodes ).
Pretty easy stuff for the "gather" phase.
The default is query_then_fetch. That said, I do not know what the
implications are for statistical facets. My assumption is that
with query_then_fetch, all calculations are done on the reducer.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
