# Track Queries with High Heap Usage **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2022, 7:00pm UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267 "2022-04-12T19:00:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2022, 7:00pm UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/1 "2022-04-12T19:00:17Z")
Hi All, I was wondering the best way to track queries based on their Heap usage. Context: I've recently had to increase heap allocation (currently 20GB RAM - 18GB Heap, up from 10GB RAM - 8GB Heap) on coordinating nodes (3 nodes in the cluster) in a cluster, as they are frequently hitting circuit breakers. This cluster has become more actively used, so while I would have expected to increase the heap a little, I wouldn't have expected to increase it this much. I suspect that potentially a few newer queries are using far more heap than I would expect, but I haven't been able to find a reliable way to track queries based on heap usage. I looked at [Slow Log](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html), but I don't think this is the correct approach, as the queries being executed aren't slow, they just (potentially) use a good amount of heap. The queries are all from Kibana features (Observability rules, SIEM rules), so I don't have too much control/ability to debug the actual queries being executed. **Setup:** Elasticsearch Version: 7.17.2 Kibana Version: 7.17.2 Install Method: Kubernetes/Containers/ECK ---
**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\ **Post date:** [April 12, 2022, 7:46pm UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/2 "2022-04-12T19:46:14Z")
Usually queries with lots of aggregation and sort tend to cause higher cpu and memory usage. What kind of queries you have in your application? Have you seen out of memory issues in any node . One way to find such queries would be to analyse the heapdumps generated on OOM scenarios. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2022, 7:53pm UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/3 "2022-04-12T19:53:29Z")
Hi @DineshNaik, the queries all come from Kibana "Rules", the main rules that we use are the `Metrics` rule under Observability, and a few `Log Threshold` rules also under observability. We also have SIEM rules (mainly the prebuilt ones), but these rules haven't really changed much since the noticeable uptick in Heap usage, so it leads me to believe that some of the `Metric` or `Log Threshold` rules have/cause an issue. Regarding OOM kills, so far, I have not noticed any OOM kills, but I have seen the coordinating nodes lock up for several minutes, near the point of getting OOM killed as the Heap takes a while to free up. ---
**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\ **Post date:** [April 13, 2022, 2:23am UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/4 "2022-04-13T02:23:54Z")
What about date ranges , has the data grown drastically and you are looking for all of it ? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 13, 2022, 12:28pm UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/5 "2022-04-13T12:28:40Z")
Regarding date ranges, most of the queries only look back the last ~5 minutes. The data has grown a bit, but not what I'd consider drastic. (Context: before the issue, cluster handled ~50k e/s, cluster now handles 65k e/s, so I wouldn't expect a doubling of heap usage requirement) ---
**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\ **Post date:** [April 14, 2022, 3:24am UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/6 "2022-04-14T03:24:29Z")
What are your compute configurations infra wise ? In slow logs have you checked if some queries are taking more time than usual? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 12, 2022, 3:24am UTC](https://discuss.elastic.co/t/track-queries-with-high-heap-usage/302267/7 "2022-05-12T03:24:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.