# Off-heap memory leak? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 16, 2018, 1:56pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202 "2018-04-16T13:56:11Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 16, 2018, 1:56pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/1 "2018-04-16T13:56:11Z")
I continue to observe an elasticsearch on docker instance that continues to grab more and more off-heap memory until Mesos kills the docker container. Since there is no data going in with the exception of the .monitor index and searches are pretty quiet on this ES instance, this is really peculiar. Note: the memory heap stays constant at just under 1GB. Any ideas? Thanks --John ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 17, 2018, 10:46am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/2 "2018-04-17T10:46:10Z")
Hi, wild guess: The ergonomic choice of the JVM for the maximum allowed direct memory is too high. Unfortunately, it is a bit hard to find out which value the JVM has chosen (you can write a small Java program which reads `sun.misc.VM.maxDirectMemory()` and invoke it in the Docker container). You can restrict direct memory explicitly e.g. to 2G with `-XX:MaxDirectMemory=2G`. Be aware that you will see more frequent garbage collections if the value you choose is too small. Also, ensure that `-XX:+DisableExplicitGC` is **not** set in your `jvm.options` (it was set by default before Elasticsearch 5.5.2). Daniel ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 17, 2018, 10:56am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/3 "2018-04-17T10:56:16Z")
Hi Daniel, Thanks for your response! Um, I believe you mean -XX:MaxDirectMemorySize=2G. 🙂 I tried that and it had no effect, with off-heap memory usage growing until Mesos killed the docker container. --John ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 17, 2018, 11:16am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/4 "2018-04-17T11:16:38Z")
Hi John, yes, I meant that parameter. Did you check what direct memory size the JVM has chosen for your container? I don't know anything about your environment and 2G might be the wrong choice as well; it was just an example how to set it. If it is really not direct memory, I'd start digging whether [native memory tracking](https://docs.oracle.com/javase/10/vm/native-memory-tracking.htm#JSJVM-GUID-710CAEA1-7C6D-4D80-AB0C-B0958E329407) reveals something. Daniel ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 17, 2018, 11:21am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/5 "2018-04-17T11:21:02Z")
Hi Daniel, Cool, and I hope my reply did not seem snarky. There are likely JVM params I am unaware of, so just wanted to ensure I am trying out what you are suggesting. Definitely great suggestion re: native memory tracking. I turned that on and restarted the container, monitoring now. Thanks, and thanks again for the follow-up! --John ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 17, 2018, 11:31am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/6 "2018-04-17T11:31:27Z")
Also, I did confirm with ps -ef that (1) the only processes running in my container are ES and the ES plugin controller and (2) the ES process is the one gobbling up more and more off-heap memory ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 17, 2018, 11:51am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/7 "2018-04-17T11:51:23Z")
Hi John, > [@hokiegeek2](#): > > Cool, and I hope my reply did not seem snarky. no, not at all. I just wanted to double-check if you might have set it to a value that is still too high. 🙂 > [@hokiegeek2](#): > > here are likely JVM params I am unaware of, so just wanted to ensure I am trying out what you are suggesting. That's not necessarily a disadvantage. Setting some exotic JVM parameter might bite you even years down the road, see [our blog](https://www.elastic.co/blog/honey-you-have-changed-quite-a-bit) for an example. I think it pays off to be conservative and adjust the defaults only when really necessary. > [@hokiegeek2](#): > > Definitely great suggestion re: native memory tracking. I turned that on and restarted the container, monitoring now. I hope that reveals something. I know that finding the root cause can be time-consuming. But often you learn interesting stuff about the JVM or the kernel. 🙂 Daniel ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 17, 2018, 12:23pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/8 "2018-04-17T12:23:10Z")
Very interesting blog post, thanks for sharing that! BTW, jvm.options...please share what your recommended settings (at least the ones you start out with). It's understood that the Xms and Xmx settings will be different. Thanks --John ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 17, 2018, 1:00pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/9 "2018-04-17T13:00:49Z")
Interesting results. I did a jcmd 1 VM.native\_memory baseline to establish a baseline and I have a script executing jcmd 1 VM.native\_memory detail.diff and writing it to a file every 2 minutes. The growth of native memory of the ES Java process is matching the increase in overall ES docker container memory usage per docker stats . The main drivers are as follows: Internal (reserved=356865KB +51292KB, committed=356861KB +51292KB) (malloc=356829KB +51292KB #25812 +390) Thread (reserved=260950KB +39298KB, committed=260950KB +39298KB) (stack: reserved=259284 +39064KB, committed=259284KB +39064KB) ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 17, 2018, 1:25pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/10 "2018-04-17T13:25:27Z")
After monitoring for 1 hour-ish, it appears that the Thread stack memory is increasing way faster than Internal. A couple of things: 1. -xss--I have this set to 1 MB (-xss1m). 2. Activity--this is a one-node cluster that has very few reads and even fewer writes. I wonder if I have long-lived threads that are not being cleaned up due to a dearth of activity and that setting the tread stack size to a relatively high value accounts for more and more off-heap being used? I further wonder if it makes sense to remove -Xssl1m would make a diff. I am gonna try that and see what happens. Definitely spit-balling here, but hey, what's the worst that can happen? 🤣 --John ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 18, 2018, 7:29am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/11 "2018-04-18T07:29:27Z")
I would not recommend reducing the thread stack size because you risk running into `StackoverflowError`. Instead, I'd recommend that you [reduce the number of threads](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-threadpool.html). To be clear: this can reduce performance but my impression is that you are more worried about memory usage than performance. Also, you should track memory usage over a longer period of time than just one hour. The application needs time to warmup properly. ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 18, 2018, 12:16pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/12 "2018-04-18T12:16:39Z")
Hi Daniel, Cool, thanks, makes sense re: thread stack size. My testing on that was not good as the off-heap memory usage actually appeared to accelerate. A co-worker of mine had his thread\_pool.generic.keep\_alive set to 30s. I set that, kept the default thread stack size by removing -xss1m and let this run overnight. Interesting. Whereas the total native memory usage went up 521MB, the Thread stack portion only went up 197MB. It appears setting keep\_alive to 30s helped and _appears_ to point to thread stacks gobbling up page cache. I restarted with a thread\_pool.generic.max set to 100 and I am monitoring. --John ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 23, 2018, 4:02pm UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/14 "2018-04-23T16:02:13Z")
Hi Daniel, Just to recap, here's where we're at configuration-wise: thread\_pool.generic.keep\_alive: 30s thread\_pool.generic.max: 100 thread\_pool.index.size: 15 thread\_pool.warmer.keep\_alive: 30s jvm.options includes the following: -XX:MaxDirectMemorySize=2G, -Xms2g, -Xmx2g And MALLOC\_ARENA\_MAX=4 At this point, ES keeps grabbing more and more off-heap memory until Mesos kills the Docker container. However, as I noted above, the percentage of RAM grabbed by the thread stack is way down from the first configuration I reported (35MB of 226278MB total, or 15%). Internal is 154MB or 68%. Any ideas as to which knob(s) to turn next? Thanks --John ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 27, 2018, 9:10am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/15 "2018-04-27T09:10:30Z")
Hi John, debugging that sort of issues is unfortunately a bit involved. Hendrik from our Cloud team has written an article about that: [Tracking Down Native Memory Leaks in Elasticsearch](https://www.elastic.co/blog/tracking-down-native-memory-leaks-in-elasticsearch). See especially the part about `jemalloc`. Daniel ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [April 27, 2018, 11:10am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/16 "2018-04-27T11:10:12Z")
Hi Daniel, Yeah, saw that article, definitely a good one. Gonna re-engage on this today and see what I can find out. Thanks again for the follow-up. --John ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 25, 2018, 11:10am UTC](https://discuss.elastic.co/t/off-heap-memory-leak/128202/17 "2018-05-25T11:10:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.