# How to prevent es cluster crash from deep aggregations **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2015, 10:00am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861 "2015-12-10T10:00:35Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![t7s](https://avatars.discourse-cdn.com/v4/letter/t/b38774/32.png) [@t7s](https://discuss.elastic.co/u/t7s)\ **Post date:** [December 10, 2015, 10:00am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/1 "2015-12-10T10:00:35Z")
Recently I have to permit a query that aggs three levels, if data is large then full gc wave is coming then the cluster is gone。 Can es server（not client） abort a request if query time is long？ ---
**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\ **Post date:** [December 10, 2015, 12:52pm UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/2 "2015-12-10T12:52:31Z")
I'm not aware of a server side timeout setting, others may chime in here. Not sure if you're aware of it already - [https://www.elastic.co/guide/en/elasticsearch/reference/2.1/circuit-breaker.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.1/circuit-breaker.html) there are memory based circuit breakers that might help you. Hope this helps, Isabel ---
**Author:** ![t7s](https://avatars.discourse-cdn.com/v4/letter/t/b38774/32.png) [@t7s](https://discuss.elastic.co/u/t7s)\ **Post date:** [December 10, 2015, 2:27pm UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/3 "2015-12-10T14:27:23Z")
indices.breaker.total.limit: 50% indices.breaker.fielddata.limit: 40% indices.fielddata.cache.size: 30% no effect, still full gc there is 20 node in my es cluster，256gb memory & 8tb ssd per node，jvm heap size 30gb per node， ES\_DIRECT\_SIZE 8gb, total data 800gb, mmap index type I found when the fuck query running the jvm size exceed my setting too much, at least 5gb more than my setting(ES\_HEAP\_SIZE: 30gb) es version is 1.5.2 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 11, 2015, 5:01am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/4 "2015-12-11T05:01:36Z")
> [@t7s](#): > > Can es server（not client） abort a request if query time is long？ No it cannot. We are going work to allow it though. ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 11, 2015, 5:28am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/5 "2015-12-11T05:28:26Z")
Why are you setting ES\_DIRECT\_SIZE? Are there other processes on the server? What version are you on? What type of data are you aggregating on? Numbers or non-analyzed strings? Are doc-values enabled (default in 2.x)? What is the cardinality of the field you are aggregating on? Tons of questions, but perhaps something might come up. Ivan ---
**Author:** ![t7s](https://avatars.discourse-cdn.com/v4/letter/t/b38774/32.png) [@t7s](https://discuss.elastic.co/u/t7s)\ **Post date:** [December 11, 2015, 6:45am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/6 "2015-12-11T06:45:11Z")
> [@Ivan](#): > > Why are you setting ES\_DIRECT\_SIZE? Are there other processes on the server? I found when the fuck query running the rss used exceed my setting too much, at least 5gb more than my setting(ES\_HEAP\_SIZE: 30gb). although I use BufferPoolMXBean to check direct memory size less than 500mb. By the way I found that if you set ES\_DIRECT\_SIZE, the cluster restart time decrease. My cluster is on 1.5.2, non-analyzed field, not cardinality but sum ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [December 11, 2015, 11:28am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/7 "2015-12-11T11:28:17Z")
I can't see your query but one thing you might want to think about is the choice between depth-first and breadth-first aggregation modes if there's a lot of interim state generated which needs to be pruned. See [https://www.elastic.co/guide/en/elasticsearch/guide/current/\_preventing\_combinatorial\_explosions.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/_preventing_combinatorial_explosions.html) ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 11, 2015, 5:39pm UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/8 "2015-12-11T17:39:40Z")
And what exactly is a fuck query? 🙂 Try enabling doc\_values on the fields that are aggregated on, but that would require a reindex. However, the ES\_DIRECT\_SIZE directive might be conflicting. Very few people, if any, should play around with ES\_DIRECT\_SIZE unless you were on a shared server. Ivan ---
**Author:** ![t7s](https://avatars.discourse-cdn.com/v4/letter/t/b38774/32.png) [@t7s](https://discuss.elastic.co/u/t7s)\ **Post date:** [December 14, 2015, 3:33am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/10 "2015-12-14T03:33:03Z")
> [@Mark\_Harwood](#): > > I can't see your query but one thing you might want to think about is the choice between depth-first and breadth-first aggregation modes if there's a lot of interim state generated which needs to be pruned. See [Preventing Combinatorial Explosions | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/_preventing_combinatorial_explosions.html) Thank you ---
**Author:** ![t7s](https://avatars.discourse-cdn.com/v4/letter/t/b38774/32.png) [@t7s](https://discuss.elastic.co/u/t7s)\ **Post date:** [December 14, 2015, 3:38am UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/11 "2015-12-14T03:38:15Z")
> [@Ivan](#): > > And what exactly is a fuck query? Get all three levels grouped result. Not me but my colleague. I know it's awful so I don't ask solution just need aborting request. Setting ES\_DIRECT\_SIZE is just an attempt，removed already ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:31pm UTC](https://discuss.elastic.co/t/how-to-prevent-es-cluster-crash-from-deep-aggregations/36861/12 "2017-07-05T23:31:24Z")