# Concerns on possible load of aggregation **URL:** \ **Category:** Elasticsearch\ **Created:** [February 25, 2015, 8:08am UTC](https://discuss.elastic.co/t/concerns-on-possible-load-of-aggregation/22375 "2015-02-25T08:08:56Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Seungjin\_Lee](https://avatars.discourse-cdn.com/v4/letter/s/2acd7d/32.png) [@Seungjin\_Lee](https://discuss.elastic.co/u/Seungjin_Lee)\ **Post date:** [February 25, 2015, 8:08am UTC](https://discuss.elastic.co/t/concerns-on-possible-load-of-aggregation/22375/1 "2015-02-25T08:08:56Z")
We are running a PAAS built with elasticsearch and we want to provide multi-column count aggregation feature through ES aggregation Let's take below as an example POST /INDEX\_PATTREN-\*/\_search { "query":{"match":{"project":"dummyProject"}}, "size":0, "aggs": { "col1": { "terms": { "field": "host", "size":5 }, "aggs": { "col2": { "terms": { "field": "source", "size":5 }, "aggs":{ "col3":{ "terms":{ "field":"version", "size":5 } } } } } } } } We use daily index, stores 30 days amount of data, approximately 500GB per day index. So the example aggreagation will investigate huge data. But we found out that it's blazingly fast, we use 20 data nodes together with several search/master nodes, and it responds within 10 minutes. OK, but what if there's many request at the same time, what can happen? Will those requests just make other requests to slow down(in this case, increase # of machines will be a solution?) or possibly cause OOM or whatever critical error on ES daemon? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL3\_U43m1UuZbAHPwSNzQHC-xpBxGsr%2B%3DGNt-GUeMCueoyTP0w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL3_U43m1UuZbAHPwSNzQHC-xpBxGsr%2B%3DGNt-GUeMCueoyTP0w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [February 25, 2015, 1:44pm UTC](https://discuss.elastic.co/t/concerns-on-possible-load-of-aggregation/22375/2 "2015-02-25T13:44:43Z")
You need to look into using an index template that uses optimal mapping for your data. For logstash, it really helps to use doc\_values on all fields you aggregate on and turning off norms as well on those fields. Doc\_values means elasticsearch uses memory mapped files instead of heap memory for the field values. WIth huge aggregations this means the system will get slower but less likely to run out of memory if you get a lot of requests. Without doc\_values, you will want to configure field data circuitbreakers properly to ensure you don't run out of memory. This typically means that searches that would have run out of memory abort with an error instead, which is preferable to your cluster crashing but not great from an end user perspective. Jilles On Wednesday, February 25, 2015 at 9:09:43 AM UTC+1, Seungjin Lee wrote: > We are running a PAAS built with elasticsearch and we want to provide > multi-column count aggregation feature through ES aggregation > > Let's take below as an example > > POST /INDEX\_PATTREN-\*/\_search > { > "query":{"match":{"project":"dummyProject"}}, > "size":0, > "aggs": { > "col1": { > "terms": { > "field": "host", > "size":5 > }, > "aggs": { > "col2": { > "terms": { > "field": "source", > "size":5 > }, > "aggs":{ > "col3":{ > "terms":{ > "field":"version", > "size":5 > } > } > } > } > } > } > } > } > > We use daily index, stores 30 days amount of data, approximately 500GB per > day index. > > So the example aggreagation will investigate huge data. > > But we found out that it's blazingly fast, we use 20 data nodes together > with several search/master nodes, and it responds within 10 minutes. > > OK, but what if there's many request at the same time, what can happen? > > Will those requests just make other requests to slow down(in this case, > increase # of machines will be a solution?) or possibly cause OOM or > whatever critical error on ES daemon? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/745a95f9-d963-472c-9ece-f326521707b5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/745a95f9-d963-472c-9ece-f326521707b5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:30am UTC](https://discuss.elastic.co/t/concerns-on-possible-load-of-aggregation/22375/3 "2017-07-06T00:30:13Z")
Closed on July 6, 2017, 12:30am UTC