# Possible memory leak in elasticsearch 6.2.4

**URL:** <https://discuss.elastic.co/t/possible-memory-leak-in-elasticsearch-6-2-4/212988>\
**Category:** Elasticsearch\
**Created:** [December 25, 2019, 2:41am UTC](https://discuss.elastic.co/t/possible-memory-leak-in-elasticsearch-6-2-4/212988 "2019-12-25T02:41:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![imperio-wxm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imperio-wxm/32/59818_2.png) [@imperio-wxm](https://discuss.elastic.co/u/imperio-wxm)\
**Post date:** [December 25, 2019, 2:41am UTC](https://discuss.elastic.co/t/possible-memory-leak-in-elasticsearch-6-2-4/212988/1 "2019-12-25T02:41:47Z")

</div>

I have a cluster for 3 nodes.

index number：2816  
shards： 24893  
docs： 1240277375  
disk use: 477G

```auto
// jvm ops
-Xms8g
-Xmx8g

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# explicitly set the stack size
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow

# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=/heap/dump/path

## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/app/rt/elasticsearch/var/logs/elasticsearch-gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=10
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/app/rt/elasticsearch/var/logs/elasticsearch-gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/d/cd9c4c439219e9736aaa987c3bb84115285a0bd5.png)

**Found that old gc does not free up lots of space in the old generation, large objects always exist in the heap.**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/d/dd05e18038523c26d021ad59804eef015612e9e4.png)

I know the jvm memory of es is divided into these parts：

1. Query cache
2. Index buffer
3. Request cache
4. Field data cache
5. Lecence memory

**However, Query cache（50MB） + Index buffer（120MB） + Request cache（50MB） + Field data cache（200KB）+ Lecence memory（460MB） = 680MB，heap use 6G，Where is the remaining 5G memory used?**

**And I get jvm dump and analyze, found out that netty takes up a lot of memory.**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fe7cc59b217f733a09752ce862273c312dbf0d15.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7ee1b30b00512d0eef9b2e705f5fb5e3d70b6cde.png)

**Provide complete analysis files if necessary.**

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [December 25, 2019, 9:27am UTC](https://discuss.elastic.co/t/possible-memory-leak-in-elasticsearch-6-2-4/212988/2 "2019-12-25T09:27:29Z")

</div>

Hi @imperio-wxm

> [@imperio-wxm](#):
>
> **And I get jvm dump and analyze, found out that netty takes up a lot of memory.**

This is expected. Netty pools memory so that it does not have to allocate new buffers for ever network read. Depending on the network IO load on an ES node this can take up a significant amount of memory, but I would not say it qualifies as a leak since that memory use does not grow boundlessly over time but rather grows and shrinks in relation to the load on the node.  
Since the buffers are pooled and hence long-lived means they count towards old-gen memory usage. You can technically disable the buffer pooling and get rid of these long lived old-gen buffers by enabling the Netty unpooled allocator but it's most likely going to result in much higher GC load and lower overall performance.

Hope that helps 🙂

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 26, 2019, 7:23am UTC](https://discuss.elastic.co/t/possible-memory-leak-in-elasticsearch-6-2-4/212988/3 "2019-12-26T07:23:46Z")

</div>

You have wwwwwaaaaayyyyyy too many shards for that size of data.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 23, 2020, 7:23am UTC](https://discuss.elastic.co/t/possible-memory-leak-in-elasticsearch-6-2-4/212988/4 "2020-01-23T07:23:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.