# Memory issue(possibly non-heap memory)

**URL:** <https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678>\
**Category:** Elasticsearch\
**Created:** [June 21, 2011, 9:01pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678 "2011-06-21T21:01:41Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhii\_2](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhii\_2](https://discuss.elastic.co/u/abhii_2)\
**Post date:** [June 21, 2011, 9:01pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/1 "2011-06-21T21:01:41Z")

</div>

I have been trying to diagnose a memory issue we are having on our  
production servers for the past couple of days. One of the current  
suspects is an elastic related issue and I need a little help to  
confirm or reject this hypothesis.

The symptoms are as follows(version details at bottom of post). The  
memory usage on our Tomcat server machine grows steadily without limit  
until all memory, both physical and swap, is exhausted, and the OS  
dies. The memory usage growth is all from the single tomcat server  
process. The fact that memory consumption grows beyond the max heap  
size points to non-heap memory growth.

The only seemingly relevant logs(currently on INFO level) are  
repeated blocks of the following:

2011-06-20 12:30:18,712 [New I/O server worker #1-6] WARN -  
[Destiny] received ping response with no matching id [84020]  
2011-06-20 12:30:18,715 [New I/O server worker #1-5] WARN -  
[Destiny] received ping response with no matching id [84020]  
2011-06-20 12:30:18,717 [New I/O server worker #1-8] WARN -  
[Destiny] received ping response with no matching id [84020]  
2011-06-20 12:30:18,727 [New I/O server worker #1-5] WARN -  
[Kala] received ping response with no matching id [84009]  
2011-06-20 12:30:18,764 [New I/O server worker #1-4] WARN -  
[Kala] received ping response with no matching id [84009]  
2011-06-20 12:30:18,770 [New I/O server worker #1-3] WARN -  
[Kala] received ping response with no matching id [84009]  
2011-06-20 12:30:18,780 [New I/O server worker #1-1] WARN -  
[Stacy, George] received ping response with no matching id [84043]  
2011-06-20 12:30:18,784 [New I/O server worker #1-2] WARN -  
[Stacy, George] received ping response with no matching id [84043]  
2011-06-20 12:30:18,793 [New I/O server worker #1-5] WARN -  
[Stacy, George] received ping response with no matching id [84043]

This happens periodically and seems to match the growth of memory  
usage(this statement needs to be verified but seems to be correct).  
It seems to happen more often initially just after the server has  
started and then at longer intervals.

Since Java NIO could allocate non-heap memory (e.g. direct ByteBuffer)  
I am wondering if anyone could confirm that this could actually be the  
cause of our leak.

PLEASE NOTE: This is on our Tomcat server machine not on our Elastic  
server machine.

The Tomcat startup command is:  
/usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/  
lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -server -  
Xss1M -Xms2G -Xmx3G -XX:NewSize=1G -XX:MaxPermSize=768M -XX:  
+UseConcMarkSweepGC -XX:+CMSIncrementalMode -  
XX:CMSInitiatingOccupancyFraction=80 -XX:+UseConcMarkSweepGC -  
Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -  
Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/  
tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -  
Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp  
org.apache.catalina.startup.Bootstrap start

The machine has 6Gb physical memory and a 700Mb swap partition. As  
you can see the -Xmx is 3G.

We are using:  
Ubuntu 2.6.32-30-server 64-bit  
Java(TM) SE Runtime Environment (build 1.6.0\_24-b07) Java HotSpot(TM)  
64-Bit Server VM (build 19.1-b02, mixed mode)  
Elastic Search 0.16.2  
Tomcat 6.0.24

Thanks a bunch.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 23, 2011, 11:41am UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/2 "2011-06-23T11:41:05Z")

</div>

Are you using Node client in the tomcat app? Which discovery are you using if this is the case, multicast or unicast?

On Wednesday, June 22, 2011 at 12:01 AM, abhii wrote:

> I have been trying to diagnose a memory issue we are having on our  
> production servers for the past couple of days. One of the current  
> suspects is an elastic related issue and I need a little help to  
> confirm or reject this hypothesis.
> 
> The symptoms are as follows(version details at bottom of post). The  
> memory usage on our Tomcat server machine grows steadily without limit  
> until all memory, both physical and swap, is exhausted, and the OS  
> dies. The memory usage growth is all from the single tomcat server  
> process. The fact that memory consumption grows beyond the max heap  
> size points to non-heap memory growth.
> 
> The only seemingly relevant logs(currently on INFO level) are  
> repeated blocks of the following:
> 
> 2011-06-20 12:30:18,712 [New I/O server worker #1-6] WARN -  
> [Destiny] received ping response with no matching id [84020]  
> 2011-06-20 12:30:18,715 [New I/O server worker #1-5] WARN -  
> [Destiny] received ping response with no matching id [84020]  
> 2011-06-20 12:30:18,717 [New I/O server worker #1-8] WARN -  
> [Destiny] received ping response with no matching id [84020]  
> 2011-06-20 12:30:18,727 [New I/O server worker #1-5] WARN -  
> [Kala] received ping response with no matching id [84009]  
> 2011-06-20 12:30:18,764 [New I/O server worker #1-4] WARN -  
> [Kala] received ping response with no matching id [84009]  
> 2011-06-20 12:30:18,770 [New I/O server worker #1-3] WARN -  
> [Kala] received ping response with no matching id [84009]  
> 2011-06-20 12:30:18,780 [New I/O server worker #1-1] WARN -  
> [Stacy, George] received ping response with no matching id [84043]  
> 2011-06-20 12:30:18,784 [New I/O server worker #1-2] WARN -  
> [Stacy, George] received ping response with no matching id [84043]  
> 2011-06-20 12:30:18,793 [New I/O server worker #1-5] WARN -  
> [Stacy, George] received ping response with no matching id [84043]
> 
> This happens periodically and seems to match the growth of memory  
> usage(this statement needs to be verified but seems to be correct).  
> It seems to happen more often initially just after the server has  
> started and then at longer intervals.
> 
> Since Java NIO could allocate non-heap memory (e.g. direct ByteBuffer)  
> I am wondering if anyone could confirm that this could actually be the  
> cause of our leak.
> 
> PLEASE NOTE: This is on our Tomcat server machine not on our Elastic  
> server machine.
> 
> The Tomcat startup command is:  
> /usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/  
> lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -server -  
> Xss1M -Xms2G -Xmx3G -XX:NewSize=1G -XX:MaxPermSize=768M -XX:  
> +UseConcMarkSweepGC -XX:+CMSIncrementalMode -  
> XX:CMSInitiatingOccupancyFraction=80 -XX:+UseConcMarkSweepGC -  
> Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -  
> Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/  
> tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -  
> Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp  
> org.apache.catalina.startup.Bootstrap start
> 
> The machine has 6Gb physical memory and a 700Mb swap partition. As  
> you can see the -Xmx is 3G.
> 
> We are using:  
> Ubuntu 2.6.32-30-server 64-bit  
> Java(TM) SE Runtime Environment (build 1.6.0\_24-b07) Java HotSpot(TM)  
> 64-Bit Server VM (build 19.1-b02, mixed mode)  
> Elastic Search 0.16.2  
> Tomcat 6.0.24
> 
> Thanks a bunch.

---

<div class="post-metadata">

**Author:** ![abhii\_2](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhii\_2](https://discuss.elastic.co/u/abhii_2)\
**Post date:** [June 23, 2011, 8:57pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/3 "2011-06-23T20:57:47Z")

</div>

The relevant code is

NodeBuilder nodeBuilder = NodeBuilder.nodeBuilder();  
Settings settings =  
nodeBuilder.settings().loadFromClasspath("elasticsearch.yml").build();  
this.elasticNode = nodeBuilder.settings(settings).client(true).node();  
this.client = this.elasticNode.client();

and the elastic.yml is ...

cluster:  
name: ssga-app.intrepid.agile

That would default to multicast right?

On Thu, Jun 23, 2011 at 4:41 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Are you using Node client in the tomcat app? Which discovery are you  
> using if this is the case, multicast or unicast?
> 
> On Wednesday, June 22, 2011 at 12:01 AM, abhii wrote:
> 
> I have been trying to diagnose a memory issue we are having on our  
> production servers for the past couple of days. One of the current  
> suspects is an elastic related issue and I need a little help to  
> confirm or reject this hypothesis.
> 
> The symptoms are as follows(version details at bottom of post). The  
> memory usage on our Tomcat server machine grows steadily without limit  
> until all memory, both physical and swap, is exhausted, and the OS  
> dies. The memory usage growth is all from the single tomcat server  
> process. The fact that memory consumption grows beyond the max heap  
> size points to non-heap memory growth.
> 
> The only seemingly relevant logs(currently on INFO level) are  
> repeated blocks of the following:
> 
> 2011-06-20 12:30:18,712 [New I/O server worker #1-6] WARN -  
> [Destiny] received ping response with no matching id [84020]  
> 2011-06-20 12:30:18,715 [New I/O server worker #1-5] WARN -  
> [Destiny] received ping response with no matching id [84020]  
> 2011-06-20 12:30:18,717 [New I/O server worker #1-8] WARN -  
> [Destiny] received ping response with no matching id [84020]  
> 2011-06-20 12:30:18,727 [New I/O server worker #1-5] WARN -  
> [Kala] received ping response with no matching id [84009]  
> 2011-06-20 12:30:18,764 [New I/O server worker #1-4] WARN -  
> [Kala] received ping response with no matching id [84009]  
> 2011-06-20 12:30:18,770 [New I/O server worker #1-3] WARN -  
> [Kala] received ping response with no matching id [84009]  
> 2011-06-20 12:30:18,780 [New I/O server worker #1-1] WARN -  
> [Stacy, George] received ping response with no matching id [84043]  
> 2011-06-20 12:30:18,784 [New I/O server worker #1-2] WARN -  
> [Stacy, George] received ping response with no matching id [84043]  
> 2011-06-20 12:30:18,793 [New I/O server worker #1-5] WARN -  
> [Stacy, George] received ping response with no matching id [84043]
> 
> This happens periodically and seems to match the growth of memory  
> usage(this statement needs to be verified but seems to be correct).  
> It seems to happen more often initially just after the server has  
> started and then at longer intervals.
> 
> Since Java NIO could allocate non-heap memory (e.g. direct ByteBuffer)  
> I am wondering if anyone could confirm that this could actually be the  
> cause of our leak.
> 
> PLEASE NOTE: This is on our Tomcat server machine not on our Elastic  
> server machine.
> 
> The Tomcat startup command is:  
> /usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/  
> lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -server -  
> Xss1M -Xms2G -Xmx3G -XX:NewSize=1G -XX:MaxPermSize=768M -XX:  
> +UseConcMarkSweepGC -XX:+CMSIncrementalMode -  
> XX:CMSInitiatingOccupancyFraction=80 -XX:+UseConcMarkSweepGC -  
> Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -  
> Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/  
> tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -  
> Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp  
> org.apache.catalina.startup.Bootstrap start
> 
> The machine has 6Gb physical memory and a 700Mb swap partition. As  
> you can see the -Xmx is 3G.
> 
> We are using:  
> Ubuntu 2.6.32-30-server 64-bit  
> Java(TM) SE Runtime Environment (build 1.6.0\_24-b07) Java HotSpot(TM)  
> 64-Bit Server VM (build 19.1-b02, mixed mode)  
> Elastic Search 0.16.2  
> Tomcat 6.0.24
> 
> Thanks a bunch.

---

<div class="post-metadata">

**Author:** ![abhii\_2](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhii\_2](https://discuss.elastic.co/u/abhii_2)\
**Post date:** [June 23, 2011, 9:22pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/4 "2011-06-23T21:22:45Z")

</div>

A few additional pieces of relevant information ...

The java code pasted above is in the init() method of a Spring bean. There  
is no other code in this init method.

Since my initial post I have been able to reproduce the fact that the ping  
response logs start appearing when the non-heap memory starts to creep up.  
To clarify, the server will be doing fine for hours and then the ping  
response logs appear and the non-heap memory starts creeping up from then  
on. However there seems to be no clear correlation between the number and  
timing of such log lines and the increase in memory. Of course, I am only  
stating that there is a clear correlation between the start of the memory  
creep and the first appearance of these ping response logs, but I have no  
idea regarding any causal link between the two.

Further, there number of threads for the tomcat process is relatively  
static, so this does not relate to runaway thread creation.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 24, 2011, 9:56am UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/5 "2011-06-24T09:56:59Z")

</div>

I don't know why the non heap is increasing, but maybe things start to garbage collect poorly causing for things to slow down. Since its running in Tomcat with "other" code, not sure if the problem is in ES or not...

On Friday, June 24, 2011 at 12:22 AM, Abhijit Inamdar wrote:

> A few additional pieces of relevant information ...
> 
> The java code pasted above is in the init() method of a Spring bean. There is no other code in this init method.
> 
> Since my initial post I have been able to reproduce the fact that the ping response logs start appearing when the non-heap memory starts to creep up. To clarify, the server will be doing fine for hours and then the ping response logs appear and the non-heap memory starts creeping up from then on. However there seems to be no clear correlation between the number and timing of such log lines and the increase in memory. Of course, I am only stating that there is a clear correlation between the start of the memory creep and the first appearance of these ping response logs, but I have no idea regarding any causal link between the two.
> 
> Further, there number of threads for the tomcat process is relatively static, so this does not relate to runaway thread creation.

---

<div class="post-metadata">

**Author:** ![kelaban](https://avatars.discourse-cdn.com/v4/letter/k/e9a140/32.png) [@kelaban](https://discuss.elastic.co/u/kelaban)\
**Post date:** [March 15, 2013, 4:39am UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/6 "2013-03-15T04:39:53Z")

</div>

I see this is a fairly old thread, but I was wondering if you ever found a fix? I am currently seeing a similar issue that is eventually leading to a OutOfMemoryError. The only difference is I am using the TransportClient

---

<div class="post-metadata">

**Author:** ![Jerome\_Gagnon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerome_gagnon/32/2178_2.png) [@Jerome\_Gagnon](https://discuss.elastic.co/u/Jerome_Gagnon)\
**Post date:** [March 15, 2013, 3:15pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/7 "2013-03-15T15:15:30Z")

</div>

I don't know the thread you are referring to, but are you using scoring  
scripts ? This is what caused me many headache due to OOM errors... I think  
there was a commit for that a while ago, it depends on which version you  
are running.

On Friday, March 15, 2013 12:39:53 AM UTC-4, Keith L wrote:

> I see this is a fairly old thread, but I was wondering if you ever found a  
> fix? I am currently seeing a similar issue that is eventually leading to a  
> OutOfMemoryError. The only difference is I am using the TransportClient
> 
> --  
> View this message in context:  
> [http://elasticsearch-users.115913.n3.nabble.com/Memory-issue-possibly-non-heap-memory-tp3092945p4031729.html](http://elasticsearch-users.115913.n3.nabble.com/Memory-issue-possibly-non-heap-memory-tp3092945p4031729.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![acv2](https://avatars.discourse-cdn.com/v4/letter/a/a4c791/32.png) [@acv2](https://discuss.elastic.co/u/acv2)\
**Post date:** [March 24, 2016, 6:28pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/8 "2016-03-24T18:28:22Z")

</div>

Hello Guys, any update on this? I'm having a similar issue i got it well documented here, but i cant figure why non heap is growing until it gets flushed and then stops everything else for some time.

here i have everything explained if you can take a look it would be great!

[http://stackoverflow.com/questions/36103454/elasticsearch-overloading-after-non-heap-mem-flush](http://stackoverflow.com/questions/36103454/elasticsearch-overloading-after-non-heap-mem-flush)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:05pm UTC](https://discuss.elastic.co/t/memory-issue-possibly-non-heap-memory/4678/9 "2017-07-05T23:05:23Z")

</div>