# Garbage Collection Not Working

**URL:** <https://discuss.elastic.co/t/garbage-collection-not-working/197173>\
**Category:** Elasticsearch\
**Created:** [August 28, 2019, 4:27pm UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173 "2019-08-28T16:27:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![stevenkrawczyk](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@stevenkrawczyk](https://discuss.elastic.co/u/stevenkrawczyk)\
**Post date:** [August 28, 2019, 4:27pm UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173/1 "2019-08-28T16:27:39Z")

</div>

Hi,

We have a 4 node cluster with "heap\_size": "29500m", and we're consistently seeing heap usage go over 80% for hours, reaching up to 99%, and no garbage collection takes place. Does anyone know why GC would not be taking place? Our current solution is to restart the elasticsearch service for each node, and then heap usage goes back to normal.

Side Note: We haven't been able to take a heap dump when usage is this high because we need this cluster to be available.

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [August 30, 2019, 10:58am UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173/2 "2019-08-30T10:58:46Z")

</div>

Hi,

please always include version information and ideally also your relevant configuration (i.e. which garbage collector are you using?). Assuming that you have garbage collection logs configured (they're [on by default since Elasticsearch 6.2.0](https://github.com/elastic/elasticsearch/pull/27610)) I suggest you inspect the GC logs. In the standard configuration the garbage collector is triggered at 75% of heap usage.

Daniel

---

<div class="post-metadata">

**Author:** ![stevenkrawczyk](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@stevenkrawczyk](https://discuss.elastic.co/u/stevenkrawczyk)\
**Post date:** [September 4, 2019, 2:42pm UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173/3 "2019-09-04T14:42:38Z")

</div>

Version is 6.6.1, and we haven't modified any settings relevant to GC, our settings (minus some names) are below.

I tried to look for GC logs but I can't find anything, I only see the regular elasticsearch logs in the logging location.

```
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.attr.rack: []
network.host:
- _local_
- _site_
bootstrap.memory_lock: true
action.destructive_requires_name: true
node.master: true
node.data: true
node.ingest: true
indices.query.bool.max_clause_count: 2048
discovery.zen.ping_timeout: 30s
discovery.zen.no_master_block: write
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts.resolve_timeout: 5s
gateway.expected_nodes: 0
gateway.expected_master_nodes: 0
gateway.expected_data_nodes: 0
gateway.recover_after_time: 5m
index.store.preload:
- "*"
index.store.type: mmapfs
thread_pool.analyze.queue_size: '200'
http.max_content_length: 500mb

```

and jvm.options

```
-Xms29500m
-Xmx29500m
-Djava.io.tmpdir=/var/data/elasticsearch/tmp
-Dlog4j2.disable.jmx=true

```

Am I correct in thinking that "Dlog4j2.disable.jmx=true" is the reason I don't see GC logs?

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 5, 2019, 11:55am UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173/4 "2019-09-05T11:55:54Z")

</div>

Hi,

GC logs are configured in `config/jvm.options` and the `jvm.options` that you've mentioned in your post above don't include any GC-related settings. Out of the box, Elasticsearch uses the CMS garbage collector. Can you please paste the output of the [nodes info API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-info.html)?

```auto
curl -s "http://YOUR_ES_HOST:9200/_nodes/jvm?pretty"

```

> [@stevenkrawczyk](#):
>
> Am I correct in thinking that "Dlog4j2.disable.jmx=true" is the reason I don't see GC logs?

[No](https://logging.apache.org/log4j/2.x/manual/jmx.html).

You need ensure that the following lines are present in `config/jvm.options` ([source](https://github.com/elastic/elasticsearch/blob/6.6/distribution/src/config/jvm.options#L104-L116)):

```auto
## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

```

After you've restarted Elasticsearch, you should see a file `gc.log` in `/var/log/elasticsearch`.

Daniel

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 10, 2019, 6:01am UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173/5 "2019-09-10T06:01:20Z")

</div>

Hi,

in the output that you've shared privately with me we can see:

```auto
        "gc_collectors" : [
          "PS Scavenge",
          "PS MarkSweep"
        ]

```

This means that the CMS garbage collector is in use. This is because you're running on JDK 8 and it is the default garbage collector there. The provided JVM options are (I've omitted any system properties):

```auto
        "input_arguments" : [
          "-Xms29500m",
          "-Xmx29500m"
        ]

```

This is _very_ minimal (even the GC is unspecified) so it appears to me you have a very non-standard `jvm.options` file. I suggest you compare your `jvm.options` file with [Elasticsearch's default `jvm.options` file for version 6.6](https://github.com/elastic/elasticsearch/blob/6.6/distribution/src/config/jvm.options) and add any missing lines. Note that that file is only a template, so you need to ensure to replace any placeholders (e.g. `${heap.dump.path}`) with proper paths on your system (see the comments in the file for guidance).

After you've done this, you should have:

- Correct system properties set (you're missing several of them, e.g. `-Dio.netty.noUnsafe=true`)
- JVM options are set so you will have garbage collection logs and also the garbage collector is explicitly configured instead of implicitly chosen by the JVM (this could bite you when upgrading the JVM).

If in doubt you can always share any files or output privately.

Daniel

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 8, 2019, 6:01am UTC](https://discuss.elastic.co/t/garbage-collection-not-working/197173/6 "2019-10-08T06:01:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.