# This query brought down my cluster :{"query": {"function\_score": {"query": {"match\_all":{}}, "random\_score": {"seed": 123456}}}} **URL:** \ **Category:** Elasticsearch\ **Created:** [December 23, 2014, 3:28am UTC](https://discuss.elastic.co/t/this-query-brought-down-my-cluster-query-function-score-query-match-all-random-score-seed-123456/21360 "2014-12-23T03:28:49Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Jinyuan\_Zhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinyuan_zhou/32/90185_2.png) [@Jinyuan\_Zhou](https://discuss.elastic.co/u/Jinyuan_Zhou)\ **Post date:** [December 23, 2014, 3:28am UTC](https://discuss.elastic.co/t/this-query-brought-down-my-cluster-query-function-score-query-match-all-random-score-seed-123456/21360/1 "2014-12-23T03:28:49Z")
Hi, I like to share my experience and in the same time hope I can get some tips. The query was run against an index with about 700 million documents. Two things happens, 1. The node run this query crashed. It is the node configured not to proccess data. 2. The data nodes start crazy on GC. eventually old generation gc cannot reduce the heep usage and the nodes becomes unresponsive. in some cases. OLD generation gc even increased size of the heap: _2014-12-20 07:21:03,370][WARN][monitor.jvm] [_\*\*\*\*_] [gc][young][2796041][224976] duration [1.1s], collections [1]/[1.3s], total [1.1s]/[3.4h], memory [21.5gb]-\>[21.2gb]/[29.8gb], all\_pools {[young] [1.4gb]-\>[3.4mb]/[1.4gb]}{[survivor] [191.3mb]-\>[191.3mb]/[191.3mb]}{[old] [19.9gb]-\>[21gb]/[28.1gb]}_ It is a bad query by itself. But I expected ES cluster handles it gracefully. It does throw this exception: - Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [\_uid] would be larger than limit of [19206989414/17.8gb]\* I guess ES stopped at some point because field data exceeds the default limit. But it is too late to stop the query that caused heap memory issue. I am wondering if there is any obvious wrong with my ES cluster configuration. I have 5 box eah with 125 ram and 32 cores. I deploy two data nodes on each of them the heap fixed at 31G and configuration is favor bulk ingesting. I actually saw above 60+K document ingesting through put per second. It was working fine until that query comes. Thanks, Jack -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jinyuan\_Zhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinyuan_zhou/32/90185_2.png) [@Jinyuan\_Zhou](https://discuss.elastic.co/u/Jinyuan_Zhou)\ **Post date:** [December 23, 2014, 7:30pm UTC](https://discuss.elastic.co/t/this-query-brought-down-my-cluster-query-function-score-query-match-all-random-score-seed-123456/21360/2 "2014-12-23T19:30:09Z")
OK, I think I miss read old generation gc issue. the line I quoted shows the old generation region mem size after a young gc. it should be expected after a young generation gc because some surviving objects are promoted from young to survival and from survivor to old. Jinyuan (Jack) Zhou On Mon, Dec 22, 2014 at 7:28 PM, Jinyuan Zhou [zhou.jinyuan@gmail.com](mailto:zhou.jinyuan@gmail.com) wrote: > Hi, > I like to share my experience and in the same time hope I can get some > tips. > > The query was run against an index with about 700 million documents. > Two things happens, > > 1. The node run this query crashed. It is the node configured not to > proccess data. > > 2. The data nodes start crazy on GC. eventually old generation gc cannot > reduce the heep usage and the nodes becomes unresponsive. in some cases. > OLD generation gc even increased size of the heap: > > _2014-12-20 07:21:03,370][WARN][monitor.jvm] [_\*\*\*\*_] > [gc][young][2796041][224976] duration [1.1s], collections [1]/[1.3s], total > [1.1s]/[3.4h], memory [21.5gb]-\>[21.2gb]/[29.8gb], all\_pools {[young] > [1.4gb]-\>[3.4mb]/[1.4gb]}{[survivor] > [191.3mb]-\>[191.3mb]/[191.3mb]}{[old] [19.9gb]-\>[21gb]/[28.1gb]}_ > > It is a bad query by itself. But I expected ES cluster handles it > gracefully. It does throw this exception: > > - Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: > [FIELDDATA] Data too large, data for [\_uid] would be larger than limit of > [19206989414 \<%5B19206989414\>/17.8gb]\* > I guess ES stopped at some point because field data exceeds the default > limit. But it is too late to stop the query that caused heap memory issue. > I am wondering if there is any obvious wrong with my ES cluster > configuration. > I have 5 box eah with 125 ram and 32 cores. I deploy two data nodes on > each of them the heap fixed at 31G and configuration is favor bulk > ingesting. I actually saw above 60+K document ingesting through put per > second. It was working fine until that query comes. > > Thanks, > > Jack > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/k2RkmjuO5OI/unsubscribe](https://groups.google.com/d/topic/elasticsearch/k2RkmjuO5OI/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANBTPCHAn78f%2BEbZ9R\_6sf5jMKYBL%3DDAZeAL2Lg\_JZV1L3peWw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANBTPCHAn78f%2BEbZ9R_6sf5jMKYBL%3DDAZeAL2Lg_JZV1L3peWw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:42am UTC](https://discuss.elastic.co/t/this-query-brought-down-my-cluster-query-function-score-query-match-all-random-score-seed-123456/21360/3 "2017-07-06T00:42:09Z")