I have a question about sub aggregations. We're using a number of terms
aggregations on some high cardinality fields, returning the top 50 results
(as set using size) in each case. We also have a cardinality
sub-aggregation on each of the terms aggregations to get the number of
unique users (a separate field) for each term returned.
We are wondering if the cardinality aggregation is executed for every
possible term found by the terms aggregation, or only the top 50 terms? We
are seeing very high memory usage and getting out of memory errors when
running this, and it's not clear from the documentation what's going on
under the hood. A cardinality aggregation on every single possible term
would go some way towards explaining things.
Is there a more efficient way of running these queries?
In the next release there is a new option to cater for this scenario. We
introduce a "collect_mode" that allows a new "breadth_first" setting on
terms aggregations which explores all of the buckets at that level and then
prunes to the top N (in your case, 50) before flowing down matches to the
child aggregations. The default mode of operation is the current
"depth_first" approach which can produce many buckets in rare cases like
the one you have encountered.
In the interim, you can do this as two requests from your client. One gets
the top 50 top-level terms then issues a second query filtered by these
selections to go get child aggs.
Cheers
Mark
On Tuesday, July 22, 2014 11:57:21 AM UTC+1, Ollie wrote:
Hi,
I have a question about sub aggregations. We're using a number of terms
aggregations on some high cardinality fields, returning the top 50 results
(as set using size) in each case. We also have a cardinality
sub-aggregation on each of the terms aggregations to get the number of
unique users (a separate field) for each term returned.
We are wondering if the cardinality aggregation is executed for every
possible term found by the terms aggregation, or only the top 50 terms? We
are seeing very high memory usage and getting out of memory errors when
running this, and it's not clear from the documentation what's going on
under the hood. A cardinality aggregation on every single possible term
would go some way towards explaining things.
Is there a more efficient way of running these queries?
Thanks for the response. That makes sense, and I've now found the relevant
bit in the documentation now –
for anyone else who's reading this discussion.
Any idea when 1.3 is likely to hit?
Ollie
On Tuesday, July 22, 2014 1:51:59 PM UTC+1, Mark Harwood wrote:
Hi Ollie,
In the next release there is a new option to cater for this scenario. We
introduce a "collect_mode" that allows a new "breadth_first" setting on
terms aggregations which explores all of the buckets at that level and then
prunes to the top N (in your case, 50) before flowing down matches to the
child aggregations. The default mode of operation is the current
"depth_first" approach which can produce many buckets in rare cases like
the one you have encountered.
In the interim, you can do this as two requests from your client. One gets
the top 50 top-level terms then issues a second query filtered by these
selections to go get child aggs.
Cheers
Mark
On Tuesday, July 22, 2014 11:57:21 AM UTC+1, Ollie wrote:
Hi,
I have a question about sub aggregations. We're using a number of terms
aggregations on some high cardinality fields, returning the top 50 results
(as set using size) in each case. We also have a cardinality
sub-aggregation on each of the terms aggregations to get the number of
unique users (a separate field) for each term returned.
We are wondering if the cardinality aggregation is executed for every
possible term found by the terms aggregation, or only the top 50 terms? We
are seeing very high memory usage and getting out of memory errors when
running this, and it's not clear from the documentation what's going on
under the hood. A cardinality aggregation on every single possible term
would go some way towards explaining things.
Is there a more efficient way of running these queries?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
