# Elasticsearch Garbage Collection issue **URL:** \ **Category:** Elasticsearch\ **Created:** [July 21, 2016, 10:19pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143 "2016-07-21T22:19:37Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [July 21, 2016, 10:19pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/1 "2016-07-21T22:19:37Z")
Elasticsearch version: 1.7.3 - 2.3.3 JVM version: "1.8.0\_72" OS version: Ubuntu 14.04 Description of the problem including expected versus actual behavior: Running ES on a 5 node cluster cluster where each node has 15GB of RAM. The max heap size is set to 7GB on each server. Using plugins searchguard-ssl, stats, and head. ![](https://us1.discourse-cdn.com/elastic/original/2X/9/93277cd40f083ef6c794ae7744f13bb6aa6a42c9.png) This is a screenshot of the heap usage on each node over a 10 day period. You can see the normal saw-tooth pattern, but the bottom of each cycle is rising. It seems that the garbage collector frees less and less memory every time that it runs. I was having a similar issue on ES 1.7.3 and hoped that bumping to 2.3.3 would help to alleviate, but no such luck. Once the servers get up to ~6GB of heap used for ES we do a rolling restart of the cluster which restarts the whole cycle. I recently posted this issue on github([https://github.com/elastic/elasticsearch/issues/19544](https://github.com/elastic/elasticsearch/issues/19544)) and was directed to the page on limiting memory usage([https://www.elastic.co/guide/en/elasticsearch/guide/current/\_limiting\_memory\_usage.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html)). The documentation leads me to believe that the field data is likely the culprit of our endlessly growing heap, however, the values for total field data used do not add up. When I run the `GET /_stats/fielddata?fields=*` on my cluster I get: ```auto "total":{ "fielddata":{ "memory_size_in_bytes":4739127224, "evictions":0, "fields":{ "@tag":{ "memory_size_in_bytes":42626416 }, "user_id":{ "memory_size_in_bytes":14236968 }, "time":{ "memory_size_in_bytes":8049808 }, "buildNum":{ "memory_size_in_bytes":816 }, "created":{ "memory_size_in_bytes":1490114912 }, "@type":{ "memory_size_in_bytes":21128028 }, "@timestamp":{ "memory_size_in_bytes":3162970276 } } } } ``` Based on these results it would seem that the total amount of memory used on field data is ~4.7GB. Based on the per node results I assume that this total is spread across all nodes in the cluster. But a segment of 4.7GB out of a total heap of 35GB (7GB per node across 5 nodes) doesn't seem like it would be the sole cause of such steep growth in our heap. Is there something else that I may be overlooking with regards to the fielddata effect on heap usage? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 21, 2016, 10:37pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/2 "2016-07-21T22:37:33Z")
Are you using doc values? ---
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [July 25, 2016, 8:53pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/3 "2016-07-25T20:53:52Z")
Hey, I have a number of old indices that are definitely not using doc\_values. My plan is to create new indices and re-index the old ones into the new ones. I understand that by default new indices will use doc\_values for all fields that are not indexed. I will post back here with the results. Is there a way that I can verify if doc\_values are being used for a given field? ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [July 25, 2016, 9:10pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/4 "2016-07-25T21:10:02Z")
> [@dopey](#): > > fields that are not indexed Fields that are not analyzed. > [@dopey](#): > > Is there a way that I can verify if doc\_values are being used for a given field? Explicitly ask for them with `"doc_values": true`. I'm not sure of another way off hand. ---
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [August 1, 2016, 7:47pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/5 "2016-08-01T19:47:25Z")
Hey, So I was definitely not using `doc_values` for the majority of my indices and documents. I didn't have templates set up for my logstash indices so most of the fields were defaulting to analyzed strings. Over the course of the last week I re-indexed all the indices in my staging environment and set up templates that would default any new `string` field to be `not_analyzed`. The re-index process itself was pretty easy - love the new `reindex` api. As a result of the re-indexing I have saved about 25% of disk space from my logstash indices (which go back ~2 months). However, I am still seeing the garbage collector reaping smaller sections of the heap with each subsequent collection. ![](https://us1.discourse-cdn.com/elastic/original/2X/c/cf72af8c56f90b21a861055a89e3590929732810.png) I've only got about 4 days worth of data, but I'm concerned because I was hoping that changing 98% of the data to be `not_analyzed` (and therefore using `doc_values` by default) would have improved these symptoms. Anything else that I may be missing? Happy to post any other cluster performance data if that would be helpful. cheers ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [August 2, 2016, 11:39am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/6 "2016-08-02T11:39:06Z")
At this point I'm not sure what it could be. The best thing might be to take a heap dump of the Java precess and crack it open in something like MAT. ---
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [August 3, 2016, 12:05am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/7 "2016-08-03T00:05:14Z")
Ive taken two heap dumps (about 7-8hrs apart) that I've imported into VisualVM. Screenshots of the histograms below (in the order that they were taken): ![](https://us1.discourse-cdn.com/elastic/original/2X/6/68e22cd9e3cc0e0232963b13bcc54f9a136b76c2.png) ![](https://us1.discourse-cdn.com/elastic/original/2X/1/1f3fb24651b43a7b5905b1f45572a9cb23961656.png) All I can tell so far is that the `byte[]` is the largest and fastest growing class in terms of size and instances. I will continue to take heap dumps over the next few days and keep comparing against the older dumps to try and locate where the largest leak is happening. Please let me know if there are any classes that I should pay particular attention to as likely suspects. cheers ---
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [August 3, 2016, 9:05pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/8 "2016-08-03T21:05:59Z")
We've taken a few more heap dumps over the last 24 hours. One thing is beginning to stand out. ![](https://us1.discourse-cdn.com/elastic/original/2X/b/bb783708bf1b98f5dcd50bfe55951c0f36506e8f.png) The taskmanager map keeps growing. It has 1.5 million instances of ReplicationTask. ![](https://us1.discourse-cdn.com/elastic/original/2X/0/083a40da5ac1ddfac2804bebb5f78b3a12576469.png) We've looked into a few of the task objects and we can see they have `phase = finished`. ![](https://us1.discourse-cdn.com/elastic/original/2X/a/a047073f943e6de57c11bf6786c6f26dbf34b5f2.png) Seems like maybe the finished tasks are not being removed, or maybe the tasks are somehow getting stuck in a weird state, and therefore not being reaped. We will keep taking heap dumps, but so far this seems a likely culprit. cheers ---
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [August 3, 2016, 10:23pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/9 "2016-08-03T22:23:18Z")
Just attempted to use the `task` api to pull down some information about running tasks and that OOM'ed my ES process (I assume because there are so many tasks in that map). Again, leading me to believe something is wonky with the task manager. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [August 3, 2016, 10:48pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/10 "2016-08-03T22:48:46Z")
> [@dopey](#): > > Again, leading me to believe something is wonky with the task manager. Yeah, that seems bad. Can you post and example of an older task? I'll open a bug and link this issue. Sorry I didn't reply to you yesterday - I'd honestly not been paying attention. Sorry! ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [August 3, 2016, 10:58pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/11 "2016-08-03T22:58:48Z")
I opened > ---
**Author:** ![dopey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dopey/32/3868_2.png) [@dopey](https://discuss.elastic.co/u/dopey)\ **Post date:** [August 4, 2016, 6:00pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/12 "2016-08-04T18:00:42Z")
Thanks for opening the issue @nik9000! Will keep updating over on github moving forwards. ---
**Author:** ![Glen\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glen_smith/32/111656_2.png) [@Glen\_Smith](https://discuss.elastic.co/u/Glen_Smith)\ **Post date:** [August 6, 2016, 5:55pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/13 "2016-08-06T17:55:55Z")
> [@nik9000](#): > > [GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine](http://github.com/elastic/elasticsearch) For the benefit of future readers: the issue was tracked down to a memory leak in a third-party plugin: search-guard-ssl. For more details, go to the github issue linked by @nik9000 above. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:29pm UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-issue/56143/14 "2017-07-05T22:29:37Z")