# Elastic Heap Size issue **URL:** \ **Category:** Elasticsearch\ **Created:** [May 28, 2018, 1:18pm UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548 "2018-05-28T13:18:08Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 28, 2018, 1:18pm UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/1 "2018-05-28T13:18:08Z")
Hi, I have elastic running on 2 nodes with 32 core, 16 core and 64 GB, 32 GB ram respectively. Heap allocated to elastic node 1 is 24 GB (node with 64 GB ram). Heap allocated to elastic node 2 is 15 GB (node with 32 GB ram). There is no segregation for Data node and Master node as of now. Currently data size is 500 GB (including both nodes) ElasticSearch Version: 2.3.0 Problem: 1. Heap memory unexpectedly starts increasing on 1 node. And never goes down until I restart that particular node. 2. Increase in Heap memory does results in increase of heap on other node, but this goes back to normal when node 1 is restarted. Actions done so far: 1. Tried deleting data reducing it to half i.e. from 1TB to 500Gb 2. Cleared field cache 3. Tried changes below parameters indices.cache.filter.size: 15% index.merge.scheduler.max\_thread\_count: 1 index.translog.flush\_threshold\_size: 1gb index.refresh\_interval: 30s indices.fielddata.cache.size: 20% indices.breaker.request.limit: 40% indices.breaker.total.limit: 70% action.auto\_create\_index: true indices.breaker.fielddata.limit: 45% Below are the runtime params being used while starting elastic on node 1 -Xms24g -Xmx24g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true Below is the output of /\_cluster/stats?pretty=1&clear=true&indices=true when heap was almost full on node 1 (64 GB RAM) { "timestamp" : 1527512323521, "cluster\_name" : "temp1", "status" : "green", "indices" : { "count" : 34, "shards" : { "total" : 148, "primaries" : 75, "replication" : 0.9733333333333334, "index" : { "shards" : { "min" : 2, "max" : 20, "avg" : 4.352941176470588 }, "primaries" : { "min" : 1, "max" : 10, "avg" : 2.2058823529411766 }, "replication" : { "min" : 0.0, "max" : 1.0, "avg" : 0.9705882352941176 } } }, "docs" : { "count" : 1214534951, "deleted" : 6776708 }, "store" : { "size\_in\_bytes" : 564568010389, "throttle\_time\_in\_millis" : 0 }, "fielddata" : { "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "query\_cache" : { "memory\_size\_in\_bytes" : 0, "total\_count" : 5958392, "hit\_count" : 132952, "miss\_count" : 5825440, "cache\_size" : 0, "cache\_count" : 14379, "evictions" : 14379 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 3840, "memory\_in\_bytes" : 2497665031, "terms\_memory\_in\_bytes" : 2233316015, "stored\_fields\_memory\_in\_bytes" : 222013960, "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory\_in\_bytes" : 696000, "doc\_values\_memory\_in\_bytes" : 41639056, "index\_writer\_memory\_in\_bytes" : 23480688, "index\_writer\_max\_memory\_in\_bytes" : 4599208143, "version\_map\_memory\_in\_bytes" : 3179664, "fixed\_bit\_set\_memory\_in\_bytes" : 0 }, "percolate" : { "total" : 0, "time\_in\_millis" : 0, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 0 } }, "nodes" : { "count" : { "total" : 2, "master\_only" : 0, "data\_only" : 0, "master\_data" : 2, "client" : 0 }, "versions" : ["2.3.0"], "os" : { "available\_processors" : 48, "allocated\_processors" : 48, "mem" : { "total\_in\_bytes" : 0 }, "names" : [ { "name" : "Linux", "count" : 2 } ] }, "process" : { "cpu" : { "percent" : 5 }, "open\_file\_descriptors" : { "min" : 3141, "max" : 3434, "avg" : 3287 } }, "jvm" : { "max\_uptime\_in\_millis" : 20946108, "versions" : [ { "version" : "1.8.0\_91", "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm\_version" : "25.91-b14", "vm\_vendor" : "Oracle Corporation", "count" : 2 } ], "mem" : { "heap\_used\_in\_bytes" : 30288790016, "heap\_max\_in\_bytes" : 41561948160 }, "threads" : 565 }, "fs" : { "total\_in\_bytes" : 2620321349632, "free\_in\_bytes" : 1478245064704, "available\_in\_bytes" : 1400792936448, "spins" : "true" }, "plugins" : [ { "name" : "cloud-aws", "version" : "2.3.0", "description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.", "jvm" : true, "classname" : "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin", "isolated" : true, "site" : false }, { "name" : "delete-by-query", "version" : "2.3.0", "description" : "The Delete By Query plugin allows to delete documents in Elasticsearch with a single query.", "jvm" : true, "classname" : "org.elasticsearch.plugin.deletebyquery.DeleteByQueryPlugin", "isolated" : true, "site" : false }, { "name" : "kopf", "version" : "2.0.1", "description" : "kopf - simple web administration tool for Elasticsearch", "url" : "/\_plugin/kopf/", "jvm" : false, "site" : true } ] } } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2018, 8:41pm UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/2 "2018-05-28T20:41:25Z")
> [@kapoorvarun23](#): > > Elasticsearch Version: 2.3.0 Can you upgrade? ---
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 29, 2018, 5:43am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/3 "2018-05-29T05:43:21Z")
@warkolm Upgrade is an option. But will it solve this issue? Don't we any other resolution for this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2018, 5:47am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/4 "2018-05-29T05:47:01Z")
Elasticsearch by default assumes all data nodes are equal so the smaller node is likely to be under more pressure. ---
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 29, 2018, 6:04am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/5 "2018-05-29T06:04:34Z")
@Christian_Dahlqvist Thanks for replying if that is the case why do my bigger node heap gets full from time to time while smaller node's heap is normal. Its not like always one of the them is going down, its random. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2018, 6:06am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/6 "2018-05-29T06:06:24Z")
Do you have monitoring installed so you can show how heap usage varies over time? > [@kapoorvarun23](#): > > heap gets full How full does it get (it is expected to get to about 75% full before GC kicks in)? Does it crash the node? ---
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 29, 2018, 6:17am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/7 "2018-05-29T06:17:23Z")
> [@Christian\_Dahlqvist](#): > > How full does it get (it is expected to get to about 75% full before GC kicks in)? Does it crash the node? It reaches above 75% and never goes back to normal until I restart elastic on that node. And yes if I wait enough to see if it goes back to normal of his own then after sometime heap gets completely full and node crashes. I can see GC running continuously in logs when heap size reaches 75% keeping that node in halt state. > [@Christian\_Dahlqvist](#): > > Do you have monitoring installed so you can show how heap usage varies over time? No monitoring installed as of now. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2018, 6:53am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/8 "2018-05-29T06:53:25Z")
I noticed that you have been overriding some of the default values. How did you arrive at these values? What happens if you stick with the defaults? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 29, 2018, 8:07am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/9 "2018-05-29T08:07:18Z")
It'll give you a better idea of what's happening, as you can leverage the Monitoring functionality in X-Pack. ---
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 29, 2018, 9:35am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/10 "2018-05-29T09:35:26Z")
> [@Christian\_Dahlqvist](#): > > I noticed that you have been overriding some of the default values. How did you arrive at these values? What happens if you stick with the defaults? I changed those values while experimenting my own debugging skills to solve this issue. This issue was occuring at default values. ---
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 29, 2018, 9:36am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/11 "2018-05-29T09:36:01Z")
> [@warkolm](#): > > It'll give you a better idea of what's happening, as you can leverage the Monitoring functionality in X-Pack. But X-Pack is paid. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 29, 2018, 9:42am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/12 "2018-05-29T09:42:48Z")
Parts of it are, but Monitoring is totally free - [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) ---
**Author:** ![kapoorvarun23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kapoorvarun23/32/31641_2.png) [@kapoorvarun23](https://discuss.elastic.co/u/kapoorvarun23)\ **Post date:** [May 30, 2018, 9:16am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/13 "2018-05-30T09:16:45Z")
Anyone? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 27, 2018, 9:16am UTC](https://discuss.elastic.co/t/elastic-heap-size-issue/133548/14 "2018-06-27T09:16:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.