# ElasticSearch Heap memory free up **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2017, 4:02am UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682 "2017-11-30T04:02:14Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 4:02am UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/1 "2017-11-30T04:02:14Z")
Hi, We are doing some load testing on ElasticSearch. Out data is log data. The testing server has 1T data, and two ElasticSearch nodes with 15g heap for each. When doing testing with some random generated queries, I noticed the heap memory is keeping increasing. And when I stop the queries, the memory is not freed (the old gen data) after about an hour. I would like to know what is the memory usage algorithm in ES or Lusence. And when would it free the old gen data? Thanks, ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 6:02pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/2 "2017-11-30T18:02:33Z")
![55 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d953ca0f74b37b91ce14a92fa7a264de92411835.png) Any one can help? ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 6:03pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/3 "2017-11-30T18:03:14Z")
![53 AM](https://us1.discourse-cdn.com/elastic/original/3X/c/3/c33f213c987d9e6f8ffb5d1181b44fb6a4280e1f.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 6:05pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/4 "2017-11-30T18:05:23Z")
It usually garbage collects once it gets to 75% used. Is there anything in the Elasticsearch logs about garbage collection? ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 6:35pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/5 "2017-11-30T18:35:32Z")
Thanks for replying. I can see gc is keep doing. The problem seems it does not able to free the old gen data. {[old] [10.7gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:19:30,680][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][586] overhead, spent [291ms] collecting in the last [1s] [2017-11-01T02:20:45,799][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][661] overhead, spent [321ms] collecting in the last [1s] [2017-11-01T02:23:15,016][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][810] overhead, spent [287ms] collecting in the last [1s] [2017-11-01T02:35:51,645][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][1566] overhead, spent [342ms] collecting in the last [1s] [2017-11-01T02:44:42,902][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2097] overhead, spent [257ms] collecting in the last [1s] [2017-11-01T02:46:53,986][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2228] overhead, spent [257ms] collecting in the last [1s] [2017-11-01T02:48:01,021][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2295] overhead, spent [265ms] collecting in the last [1s] [2017-11-01T02:48:02,034][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2296] overhead, spent [262ms] collecting in the last [1s] [2017-11-01T02:53:44,505][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2637] overhead, spent [289ms] collecting in the last [1s] [2017-11-01T02:55:08,319][INFO][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2702][18] duration [18s], collections [2]/[18.9s], total [18s]/[19.8s], memory [11.5gb]-\>[10.9gb]/[11.8gb], all\_pools {[young] [696.4mb]-\>[14.4mb]/[865.3mb]}{[survivor] [108.1mb]-\>[0b]/[108.1mb]}{[old] [10.7gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:55:08,319][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2702] overhead, spent [18s] collecting in the last [18.9s] [2017-11-01T02:55:32,287][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2704][19] duration [21.9s], collections [1]/[22.9s], total [21.9s]/[41.8s], memory [11.3gb]-\>[11gb]/[11.8gb], all\_pools {[young] [476.7mb]-\>[119.3mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:55:32,287][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2704] overhead, spent [21.9s] collecting in the last [22.9s] [2017-11-01T02:55:50,875][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2705][20] duration [18.2s], collections [1]/[18.5s], total [18.2s]/[1m], memory [11gb]-\>[11.2gb]/[11.8gb], all\_pools {[young] [119.3mb]-\>[278.7mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:55:50,888][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2705] overhead, spent [18.2s] collecting in the last [18.5s] [2017-11-01T02:56:12,513][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2706][21] duration [21.3s], collections [1]/[21.6s], total [21.3s]/[1.3m], memory [11.2gb]-\>[11gb]/[11.8gb], all\_pools {[young] [278.7mb]-\>[80.1mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:56:12,513][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2706] overhead, spent [21.3s] collecting in the last [21.6s] [2017-11-01T02:56:30,924][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2707][22] duration [17.9s], collections [1]/[18.4s], total [17.9s]/[1.6m], memory [11gb]-\>[10.9gb]/[11.8gb], all\_pools {[young] [80.1mb]-\>[40.6mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:56:30,924][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2707] overhead, spent [17.9s] collecting in the last [18.4s] [2017-11-01T02:56:52,476][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2708][23] duration [21.2s], collections [1]/[21.5s], total [21.2s]/[2m], memory [10.9gb]-\>[11gb]/[11.8gb], all\_pools {[young] [40.6mb]-\>[62.2mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:56:52,487][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2708] overhead, spent [21.2s] collecting in the last [21.5s] [2017-11-01T02:57:10,696][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2709][24] duration [17.8s], collections [1]/[18.2s], total [17.8s]/[2.3m], memory [11gb]-\>[10.9gb]/[11.8gb], all\_pools {[young] [62.2mb]-\>[28.6mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:57:10,696][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2709] overhead, spent [17.8s] collecting in the last [18.2s] [2017-11-01T02:57:31,420][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][old][2710][25] duration [20.2s], collections [1]/[20.7s], total [20.2s]/[2.6m], memory [10.9gb]-\>[10.9gb]/[11.8gb], all\_pools {[young] [28.6mb]-\>[54.2mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [10.9gb]-\>[10.9gb]/[10.9gb]} [2017-11-01T02:57:31,420][WARN][o.e.m.j.JvmGcMonitorService] [host1-1] [gc][2710] overhead, spent [20.2s] collecting in the last [20.7s] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 6:37pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/6 "2017-11-30T18:37:29Z")
What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/cluster-stats.html)? ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 6:45pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/7 "2017-11-30T18:45:31Z")
I have the node status result: "\_shards": { "total": 1157, "successful": 1157, "failed": 0 }, "\_all": { "primaries": { "docs": { "count": 1124663379, "deleted": 0 }, "store": { "size\_in\_bytes": 1222205430485, "throttle\_time\_in\_millis": 0 }, "indexing": { "index\_total": 0, "index\_time\_in\_millis": 0, "index\_current": 0, "index\_failed": 0, "delete\_total": 0, "delete\_time\_in\_millis": 0, "delete\_current": 0, "noop\_update\_total": 0, "is\_throttled": false, "throttle\_time\_in\_millis": 0 }, "get": { "total": 4, "time\_in\_millis": 6, "exists\_total": 4, "exists\_time\_in\_millis": 6, "missing\_total": 0, "missing\_time\_in\_millis": 0, "current": 0 }, "search": { "open\_contexts": 0, "query\_total": 27582, "query\_time\_in\_millis": 1973771, "query\_current": 0, "fetch\_total": 27582, "fetch\_time\_in\_millis": 3205556, "fetch\_current": 0, "scroll\_total": 0, "scroll\_time\_in\_millis": 0, "scroll\_current": 0, "suggest\_total": 0, "suggest\_time\_in\_millis": 0, "suggest\_current": 0 }, "merges": { "current": 0, "current\_docs": 0, "current\_size\_in\_bytes": 0, "total": 0, "total\_time\_in\_millis": 0, "total\_docs": 0, "total\_size\_in\_bytes": 0, "total\_stopped\_time\_in\_millis": 0, "total\_throttled\_time\_in\_millis": 0, "total\_auto\_throttle\_in\_bytes": 24243077120 }, "refresh": { "total": 1156, "total\_time\_in\_millis": 28470, "listeners": 0 }, "flush": { "total": 1156, "total\_time\_in\_millis": 70 }, "warmer": { "current": 0, "total": 2312, "total\_time\_in\_millis": 2866 }, "query\_cache": { "memory\_size\_in\_bytes": 273355040, "total\_count": 449953, "hit\_count": 160211, "miss\_count": 289742, "cache\_size": 27168, "cache\_count": 27168, "evictions": 0 }, "fielddata": { "memory\_size\_in\_bytes": 0, "evictions": 0 }, "completion": { "size\_in\_bytes": 0 }, "segments": { "count": 22998, "memory\_in\_bytes": 3395618190, "terms\_memory\_in\_bytes": 2422319263, "stored\_fields\_memory\_in\_bytes": 109814856, "term\_vectors\_memory\_in\_bytes": 0, "norms\_memory\_in\_bytes": 274366848, "points\_memory\_in\_bytes": 41837863, "doc\_values\_memory\_in\_bytes": 547279360, "index\_writer\_memory\_in\_bytes": 0, "version\_map\_memory\_in\_bytes": 0, "fixed\_bit\_set\_memory\_in\_bytes": 0, "max\_unsafe\_auto\_id\_timestamp": -1, "file\_sizes": {} }, "translog": { "operations": 0, "size\_in\_bytes": 99416 }, "request\_cache": { "memory\_size\_in\_bytes": 0, "evictions": 0, "hit\_count": 0, "miss\_count": 0 }, "recovery": { "current\_as\_source": 0, "current\_as\_target": 0, "throttle\_time\_in\_millis": 0 } }, "total": { "docs": { "count": 1124663405, "deleted": 0 }, "store": { "size\_in\_bytes": 1222205525166, "throttle\_time\_in\_millis": 0 }, "indexing": { "index\_total": 0, "index\_time\_in\_millis": 0, "index\_current": 0, "index\_failed": 0, "delete\_total": 0, "delete\_time\_in\_millis": 0, "delete\_current": 0, "noop\_update\_total": 0, "is\_throttled": false, "throttle\_time\_in\_millis": 0 }, "get": { "total": 10, "time\_in\_millis": 11, "exists\_total": 10, "exists\_time\_in\_millis": 11, "missing\_total": 0, "missing\_time\_in\_millis": 0, "current": 0 }, "search": { "open\_contexts": 0, "query\_total": 28084, "query\_time\_in\_millis": 1973872, "query\_current": 0, "fetch\_total": 28084, "fetch\_time\_in\_millis": 3205612, "fetch\_current": 0, "scroll\_total": 0, "scroll\_time\_in\_millis": 0, "scroll\_current": 0, "suggest\_total": 0, "suggest\_time\_in\_millis": 0, "suggest\_current": 0 }, "merges": { "current": 0, "current\_docs": 0, "current\_size\_in\_bytes": 0, "total": 0, "total\_time\_in\_millis": 0, "total\_docs": 0, "total\_size\_in\_bytes": 0, "total\_stopped\_time\_in\_millis": 0, "total\_throttled\_time\_in\_millis": 0, "total\_auto\_throttle\_in\_bytes": 24264048640 }, "refresh": { "total": 1157, "total\_time\_in\_millis": 28472, "listeners": 0 }, "flush": { "total": 1157, "total\_time\_in\_millis": 70 }, "warmer": { "current": 0, "total": 2314, "total\_time\_in\_millis": 2866 }, "query\_cache": { "memory\_size\_in\_bytes": 273355040, "total\_count": 449953, "hit\_count": 160211, "miss\_count": 289742, "cache\_size": 27168, "cache\_count": 27168, "evictions": 0 }, "fielddata": { "memory\_size\_in\_bytes": 0, "evictions": 0 }, "completion": { "size\_in\_bytes": 0 }, "segments": { "count": 23004, "memory\_in\_bytes": 3395645708, "terms\_memory\_in\_bytes": 2422340287, "stored\_fields\_memory\_in\_bytes": 109816728, "term\_vectors\_memory\_in\_bytes": 0, "norms\_memory\_in\_bytes": 274369280, "points\_memory\_in\_bytes": 41837877, "doc\_values\_memory\_in\_bytes": 547281536, "index\_writer\_memory\_in\_bytes": 0, "version\_map\_memory\_in\_bytes": 0, "fixed\_bit\_set\_memory\_in\_bytes": 0, "max\_unsafe\_auto\_id\_timestamp": -1, "file\_sizes": {} }, "translog": { "operations": 0, "size\_in\_bytes": 99459 }, "request\_cache": { "memory\_size\_in\_bytes": 0, "evictions": 0, "hit\_count": 0, "miss\_count": 0 }, "recovery": { "current\_as\_source": 0, "current\_as\_target": 0, "throttle\_time\_in\_millis": 0 } } }, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 6:59pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/8 "2017-11-30T18:59:51Z")
That does not seem to be the full output. Can you provide the full response in a Gist (if it does not fit here)? ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 7:02pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/9 "2017-11-30T19:02:34Z")
> ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 7:04pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/10 "2017-11-30T19:04:54Z")
From what I can see from the part above, it looks like you have a lot of quite small shards, resulting in a lot of small segments. You may want to read this [blog post on shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for some guidance. ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 8:12pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/11 "2017-11-30T20:12:02Z")
Thanks, this is helpful. But do you mean the shards info in taking the old gen space and never get free? I have the cluster health info here, hope it can reveal more. > ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 8:28pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/12 "2017-11-30T20:28:34Z")
As outlined in the blog post I linked to, each shard ahs some memory overhead that uses heap. By reducing the number of shards and segments you should be able to reduce heap usage. At the moment it seems like it is near the limit and not able to free much when it does GC. ---
**Author:** ![akirafei](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@akirafei](https://discuss.elastic.co/u/akirafei)\ **Post date:** [November 30, 2017, 10:41pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/13 "2017-11-30T22:41:35Z")
Thanks Christian, we will try to improve it. BTW, is there a way to get the detailed heap usage for elasticseach? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 28, 2017, 10:41pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-memory-free-up/109682/14 "2017-12-28T22:41:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.