# Heap usage of client node is continuously increasing as I run queries **URL:** \ **Category:** Elasticsearch\ **Created:** [March 1, 2017, 5:14pm UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032 "2017-03-01T17:14:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [March 1, 2017, 5:14pm UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032/1 "2017-03-01T17:14:51Z")
ES 5.2.2, Ubuntu 16.04, Oracle JRE8 I have a 6x m4.large nodes, 120 M docs split into nearly 70 indices with 5 shards, 1 replica. Querying is only done on latest data about 1 M docs using time filters. This is a test cluster with no real traffic. request\_cache is only about 5-10 MB when the heap usage is at around 3 GB. When I simulate a couple of queries [https://gist.github.com/vanga/2cd8e1fd7c3b2bffa89fda8ce3a8a481](https://gist.github.com/vanga/2cd8e1fd7c3b2bffa89fda8ce3a8a481), I see the increase in heap usage till the point old GC runs , heap comes back to normal after this, but with real traffic, having frequent old GCs is really making the cluster unusable. This increase is only on the node I am making the request to. I have also force merged segments to max\_num\_segments=2. Field data is not significant. Any leads on how to debug this further would be helpful. Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2017, 11:23pm UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032/2 "2017-03-01T23:23:26Z")
Do you have Monitoring installed? If not, starting there would be a good idea as you can then see what's going on with a bit more detail. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [March 2, 2017, 1:19am UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032/3 "2017-03-02T01:19:40Z")
I have monitoring installed, but there aren't lot of metrics. Only heap usage metric is relevant here I think. I checked the other metrics from node\_stats API I will just collate the observations at one place Heap is increasing, Old GCs are happening frequently Request cache, query cache and fielddata are all in terms of few MBs Segment count to max\_num\_segments=2, data is constant Heap usage is only increasing on the node that I am making request. Please let me know if you have any specific metrics in mind, I can take a look at them. Thanks. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [March 2, 2017, 2:28am UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032/4 "2017-03-02T02:28:08Z")
![](https://us1.discourse-cdn.com/elastic/original/2X/b/b7c1c929e286b877d3640fedc07b8e9de8b8e63a.png) Here is the node status after running the two queries in the gist every 2 seconds. Note, Though the data is constant, the queries may not be exactly same in each run. order of terms list might be changing in each run as the queries are auto generated. I have also updated the gist with the output of node stats. Thanks. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [March 3, 2017, 6:05am UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032/5 "2017-03-03T06:05:33Z")
I am not an expert with JVM/GCs, but here are my observations. I notice that as soon as I run a query, Old memory pool size is increasing. Isn't it strange that objects get moved to the old pool so fast? is there anything in ES that forces the objects to be moved to old GC straight away (I believe it's JVM that handles this memory management and ES has no control?) What I did: I restarted my node, noted different memory pools sizes, ran the query and saw that old pool size has increased. There is no other traffic other than the query I made. Is something like increasing the young pool size a solution? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 31, 2017, 6:05am UTC](https://discuss.elastic.co/t/heap-usage-of-client-node-is-continuously-increasing-as-i-run-queries/77032/6 "2017-03-31T06:05:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.