# Memory problems

**URL:** <https://discuss.elastic.co/t/memory-problems/4661>\
**Category:** Elasticsearch\
**Created:** [June 20, 2011, 6:25pm UTC](https://discuss.elastic.co/t/memory-problems/4661 "2011-06-20T18:25:36Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![Mathew\_Davies](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mathew_davies/32/3167_2.png) [@Mathew\_Davies](https://discuss.elastic.co/u/Mathew_Davies)\
**Post date:** [July 13, 2011, 4:14pm UTC](https://discuss.elastic.co/t/memory-problems/4661/21 "2011-07-13T16:14:10Z")

</div>

Just to let you know, Kimchy has tracked down and fixed a memory leak. You  
can see the commit here:

> <https://github.com/elastic/elasticsearch/commit/1033249f0c971ea099ccbce6d9fe3cd720c7c800>

If people can test this out and let us know if it works, that'd be great.

Kind Regards,  
-Mathew Davies.

On 8 July 2011 01:59, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> Its possible, I am trying to track this down and check.
> 
> On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> 
> We are not using sigar directly, unless ES does something with it. We  
> are running ES as a service.
> 
> Is it possible anything in the network layer or nio stuff could be  
> leaking?
> 
> Thanks so much,  
> Lar
> 
> On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> Thats a difficult one the figure out. elasticsearch does not use native  
> memory (by default), and this leak can come from allocations of native  
> memory, bugs in native libraries used, or leaks in the java process itself.  
> Are you using sigar within that app?
> 
> On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> 
> I'm not sure I follow your suggestion: As I mentioned, the memory  
> leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> monitor shows normal memory usage and garbage collection patterns.  
> However the java process memory grows until all system memory is gone.
> 
> On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> 
> Try to find out which kind ofmemorythe heap is piling up (Perma Old New).  
> Tomcat is a notorious PermaGen leaker.
> 
> -------- Original Message --------  
> Subject: Re:Memoryproblems  
> From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre   
> [...@gmail.com](mailto:...@gmail.com))\>  
> Date: Wed, July 06, 2011 4:52 pm  
> To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us [...@elasticsearch.com](mailto:...@elasticsearch.com))\>  
> We are seeing the same issue. That is, running elasticsearch server  
> on linux with the sun jdk, the applicationmemoryseems constant, but  
> the swapmemorysteadily grows over the course of a week or so, until  
> we run out of swap space.  
> Additionally, I believe that our elastic client that runs in a Tomcat  
> webapp is leaking non-heapmemory. After a period of time, the  
> processmemory(but not the JVMmemory) starts to grow steadily until  
> we have to restart the tomcat server. We never get a java "out ofmemory"  
> error, instead the java process eventually consumes all of  
> systemmemory.  
> We have been doing our best to isolate this, and it really does seem  
> to be the elastic client that is leaking thememory. Perhaps the  
> sever side swapmemorygrowth is related.  
> I think we need help with this one.  
> Thanks,  
> Lar

---

<div class="post-metadata">

**Author:** ![Mathew\_Davies](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mathew_davies/32/3167_2.png) [@Mathew\_Davies](https://discuss.elastic.co/u/Mathew_Davies)\
**Post date:** [July 14, 2011, 4:49pm UTC](https://discuss.elastic.co/t/memory-problems/4661/22 "2011-07-14T16:49:17Z")

</div>

20 hours have past and I can confirm that Shay has fixed the memory leak.

See my new munin graph here: [http://i.imgur.com/MCGW0.png](http://i.imgur.com/MCGW0.png)

If you can't upgrade your servers to use the latest code from the repository  
you can also do what Vertice suggested [1] and pass the

Des.monitor.jvm.enabled=false

flag to the binary.

[1]

> <https://github.com/elastic/elasticsearch/issues/1075#issuecomment-1571469>
>
> I had a long discussion about this on the mailing list. The solutions tried have… had no effect so I'm considering this a bug.
> \## System Information
> \- Total memory = 469M
> \- vm.swappiness = 10
> \### Java
> 
> \`\`\` bash
> Vendor: Sun Microsystems Inc.
> Name: Java HotSpot(TM) Client VM
> Version: 19.1-b02
> Java version: 1.6.0\_24
> \`\`\`
> \### Elastic
> 
> \`\`\` bash
> Version: 0.16.2
> \`\`\`
> \#### Indicies
> 
> \`\`\` bash
> Number of documents: 3888, Store size: 1.8mb (1925915 B)
> Field cache evictions: 0, Field cache size: 125.9kb, Filter cache size: 18.3kb
> Merges: Current: 0, Total: 0, Took: 0s
> \`\`\`
> \### Graphs
> 
> !\[\](http://localhostr.com/file/LDs4Yad/localhost.localdomain-memory-day.png)
> !\[\](http://localhostr.com/file/O6YDxPU/localhost.localdomain-memory-week.png)
> !\[\](http://localhostr.com/file/uypPCsE/localhost.localdomain-memory-month.png)
> 
> \<a href="http://localhostr.com/file/6xvX3fN/Selection\_029.jpeg"\>\<img src="http://localhostr.com/file/6xvX3fN/Selection\_029.jpeg" width="495" /\>\</a\>
> \## Solutions tried
> \### Locking memory
> 
> No effect. The following option was set for the elastisearch user: 
> 
> \`\`\`
> ulimit -l unlimited
> \`\`\`
> \### Swapped from OpenJDK to Oracle's Java
> 
> No effect
> \### Next action
> 
> I'll be trying to monitor the JVM at a lower level using a tool like visualgc to see what I can gain from it.

Kind Regards,  
-Mathew Davies.

On 13 July 2011 17:14, Mathew Davies [thepixeldeveloper@googlemail.com](mailto:thepixeldeveloper@googlemail.com)wrote:

> Just to let you know, Kimchy has tracked down and fixed a memory leak. You  
> can see the commit here:
> 
> [Native (java) process memory leak, closes #1118. · elastic/elasticsearch@1033249 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/1033249f0c971ea099ccbce6d9fe3cd720c7c800)
> 
> If people can test this out and let us know if it works, that'd be great.
> 
> Kind Regards,  
> -Mathew Davies.
> 
> On 8 July 2011 01:59, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > Its possible, I am trying to track this down and check.
> > 
> > On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> > 
> > We are not using sigar directly, unless ES does something with it. We  
> > are running ES as a service.
> > 
> > Is it possible anything in the network layer or nio stuff could be  
> > leaking?
> > 
> > Thanks so much,  
> > Lar
> > 
> > On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > Thats a difficult one the figure out. elasticsearch does not use native  
> > memory (by default), and this leak can come from allocations of native  
> > memory, bugs in native libraries used, or leaks in the java process itself.  
> > Are you using sigar within that app?
> > 
> > On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> > 
> > I'm not sure I follow your suggestion: As I mentioned, the memory  
> > leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> > monitor shows normal memory usage and garbage collection patterns.  
> > However the java process memory grows until all system memory is gone.
> > 
> > On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> > 
> > Try to find out which kind ofmemorythe heap is piling up (Perma Old New).  
> > Tomcat is a notorious PermaGen leaker.
> > 
> > -------- Original Message --------  
> > Subject: Re:Memoryproblems  
> > From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre   
> > [...@gmail.com](mailto:...@gmail.com))\>  
> > Date: Wed, July 06, 2011 4:52 pm  
> > To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us [...@elasticsearch.com](mailto:...@elasticsearch.com))\>  
> > We are seeing the same issue. That is, running elasticsearch server  
> > on linux with the sun jdk, the applicationmemoryseems constant, but  
> > the swapmemorysteadily grows over the course of a week or so, until  
> > we run out of swap space.  
> > Additionally, I believe that our elastic client that runs in a Tomcat  
> > webapp is leaking non-heapmemory. After a period of time, the  
> > processmemory(but not the JVMmemory) starts to grow steadily until  
> > we have to restart the tomcat server. We never get a java "out ofmemory"  
> > error, instead the java process eventually consumes all of  
> > systemmemory.  
> > We have been doing our best to isolate this, and it really does seem  
> > to be the elastic client that is leaking thememory. Perhaps the  
> > sever side swapmemorygrowth is related.  
> > I think we need help with this one.  
> > Thanks,  
> > Lar

---

<div class="post-metadata">

**Author:** ![Lar\_Mader](https://avatars.discourse-cdn.com/v4/letter/l/a6a055/32.png) [@Lar\_Mader](https://discuss.elastic.co/u/Lar_Mader)\
**Post date:** [July 18, 2011, 11:25pm UTC](https://discuss.elastic.co/t/memory-problems/4661/23 "2011-07-18T23:25:55Z")

</div>

Could this leak have affected the client as well?

Thanks,  
Lar

On Jul 14, 9:49 am, Mathew Davies [thepixeldevelo...@googlemail.com](mailto:thepixeldevelo...@googlemail.com)  
wrote:

> 20 hours have past and I can confirm that Shay has fixed the memory leak.
> 
> See my new munin graph here:[http://i.imgur.com/MCGW0.png](http://i.imgur.com/MCGW0.png)
> 
> If you can't upgrade your servers to use the latest code from the repository  
> you can also do what Vertice suggested [1] and pass the
> 
> Des.monitor.jvm.enabled=false
> 
> flag to the binary.
> 
> [1][Memory leak · Issue #1075 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1075#issuecomme)...
> 
> Kind Regards,  
> -Mathew Davies.
> 
> On 13 July 2011 17:14, Mathew Davies [thepixeldevelo...@googlemail.com](mailto:thepixeldevelo...@googlemail.com)wrote:
> 
> > Just to let you know, Kimchy has tracked down and fixed a memory leak. You  
> > can see the commit here:
> 
> > [Native (java) process memory leak, closes #1118. · elastic/elasticsearch@1033249 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/1033249f0c971ea)...
> 
> > If people can test this out and let us know if it works, that'd be great.
> 
> > Kind Regards,  
> > -Mathew Davies.
> 
> > On 8 July 2011 01:59, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > Its possible, I am trying to track this down and check.
> 
> > > On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> 
> > > We are not using sigar directly, unless ES does something with it. We  
> > > are running ES as a service.
> 
> > > Is it possible anything in the network layer or nio stuff could be  
> > > leaking?
> 
> > > Thanks so much,  
> > > Lar
> 
> > > On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > Thats a difficult one the figure out. elasticsearch does not use native  
> > > memory (by default), and this leak can come from allocations of native  
> > > memory, bugs in native libraries used, or leaks in the java process itself.  
> > > Are you using sigar within that app?
> 
> > > On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> 
> > > I'm not sure I follow your suggestion: As I mentioned, the memory  
> > > leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> > > monitor shows normal memory usage and garbage collection patterns.  
> > > However the java process memory grows until all system memory is gone.
> 
> > > On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> 
> > > Try to find out which kind ofmemorythe heap is piling up (Perma Old New).  
> > > Tomcat is a notorious PermaGen leaker.
> 
> > > -------- Original Message --------  
> > > Subject: Re:Memoryproblems  
> > > From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre   
> > > [....@gmail.com](mailto:....@gmail.com))\>  
> > > Date: Wed, July 06, 2011 4:52 pm  
> > > To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us [....@elasticsearch.com](mailto:....@elasticsearch.com))\>  
> > > We are seeing the same issue. That is, running elasticsearch server  
> > > on linux with the sun jdk, the applicationmemoryseems constant, but  
> > > the swapmemorysteadily grows over the course of a week or so, until  
> > > we run out of swap space.  
> > > Additionally, I believe that our elastic client that runs in a Tomcat  
> > > webapp is leaking non-heapmemory. After a period of time, the  
> > > processmemory(but not the JVMmemory) starts to grow steadily until  
> > > we have to restart the tomcat server. We never get a java "out ofmemory"  
> > > error, instead the java process eventually consumes all of  
> > > systemmemory.  
> > > We have been doing our best to isolate this, and it really does seem  
> > > to be the elastic client that is leaking thememory. Perhaps the  
> > > sever side swapmemorygrowth is related.  
> > > I think we need help with this one.  
> > > Thanks,  
> > > Lar- Hide quoted text -
> 
> - Show quoted text -

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 18, 2011, 11:29pm UTC](https://discuss.elastic.co/t/memory-problems/4661/24 "2011-07-18T23:29:57Z")

</div>

Yes, since the scheduled jvm monitor logger that uses the offending java  
call is also started when using a node client. Not so, btw, when using  
transport client.

On Tue, Jul 19, 2011 at 2:25 AM, lmader [lmaderintrepid@gmail.com](mailto:lmaderintrepid@gmail.com) wrote:

> Could this leak have affected the client as well?
> 
> Thanks,  
> Lar
> 
> On Jul 14, 9:49 am, Mathew Davies [thepixeldevelo...@googlemail.com](mailto:thepixeldevelo...@googlemail.com)  
> wrote:
> 
> > 20 hours have past and I can confirm that Shay has fixed the memory leak.
> > 
> > See my new munin graph here:[http://i.imgur.com/MCGW0.png](http://i.imgur.com/MCGW0.png)
> > 
> > If you can't upgrade your servers to use the latest code from the  
> > repository  
> > you can also do what Vertice suggested [1] and pass the
> > 
> > Des.monitor.jvm.enabled=false
> > 
> > flag to the binary.
> > 
> > [1]  
> > [Memory leak · Issue #1075 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1075#issuecomme)...
> > 
> > Kind Regards,  
> > -Mathew Davies.
> > 
> > On 13 July 2011 17:14, Mathew Davies \<[thepixeldevelo...@googlemail.com](mailto:thepixeldevelo...@googlemail.com)  
> > wrote:
> > 
> > > Just to let you know, Kimchy has tracked down and fixed a memory leak.  
> > > You  
> > > can see the commit here:
> > 
> > > [Native (java) process memory leak, closes #1118. · elastic/elasticsearch@1033249 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/1033249f0c971ea).  
> > > ..
> > 
> > > If people can test this out and let us know if it works, that'd be  
> > > great.
> > 
> > > Kind Regards,  
> > > -Mathew Davies.
> > 
> > > On 8 July 2011 01:59, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > Its possible, I am trying to track this down and check.
> > 
> > > > On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> > 
> > > > We are not using sigar directly, unless ES does something with it. We  
> > > > are running ES as a service.
> > 
> > > > Is it possible anything in the network layer or nio stuff could be  
> > > > leaking?
> > 
> > > > Thanks so much,  
> > > > Lar
> > 
> > > > On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > Thats a difficult one the figure out. elasticsearch does not use  
> > > > native  
> > > > memory (by default), and this leak can come from allocations of native  
> > > > memory, bugs in native libraries used, or leaks in the java process  
> > > > itself.  
> > > > Are you using sigar within that app?
> > 
> > > > On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> > 
> > > > I'm not sure I follow your suggestion: As I mentioned, the memory  
> > > > leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> > > > monitor shows normal memory usage and garbage collection patterns.  
> > > > However the java process memory grows until all system memory is gone.
> > 
> > > > On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> > 
> > > > Try to find out which kind ofmemorythe heap is piling up (Perma Old  
> > > > New).  
> > > > Tomcat is a notorious PermaGen leaker.
> > 
> > > > -------- Original Message --------  
> > > > Subject: Re:Memoryproblems  
> > > > From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre  
> > > > [....@gmail.com](mailto:....@gmail.com))\>  
> > > > Date: Wed, July 06, 2011 4:52 pm  
> > > > To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us ....@  
> > > > [elasticsearch.com](http://elasticsearch.com))\>  
> > > > We are seeing the same issue. That is, running elasticsearch server  
> > > > on linux with the sun jdk, the applicationmemoryseems constant, but  
> > > > the swapmemorysteadily grows over the course of a week or so, until  
> > > > we run out of swap space.  
> > > > Additionally, I believe that our elastic client that runs in a Tomcat  
> > > > webapp is leaking non-heapmemory. After a period of time, the  
> > > > processmemory(but not the JVMmemory) starts to grow steadily until  
> > > > we have to restart the tomcat server. We never get a java "out  
> > > > ofmemory"  
> > > > error, instead the java process eventually consumes all of  
> > > > systemmemory.  
> > > > We have been doing our best to isolate this, and it really does seem  
> > > > to be the elastic client that is leaking thememory. Perhaps the  
> > > > sever side swapmemorygrowth is related.  
> > > > I think we need help with this one.  
> > > > Thanks,  
> > > > Lar- Hide quoted text -
> > 
> > - Show quoted text -

---

<div class="post-metadata">

**Author:** ![Lar\_Mader](https://avatars.discourse-cdn.com/v4/letter/l/a6a055/32.png) [@Lar\_Mader](https://discuss.elastic.co/u/Lar_Mader)\
**Post date:** [July 18, 2011, 11:34pm UTC](https://discuss.elastic.co/t/memory-problems/4661/25 "2011-07-18T23:34:01Z")

</div>

Ok, that's us (we're using the node client). This sounds extremely  
promising. We'll test out the latest release and let you know if this  
solves it for us.

As always, thanks so much!  
Lar

On Jul 18, 4:29 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Yes, since the scheduled jvm monitor logger that uses the offending java  
> call is also started when using a node client. Not so, btw, when using  
> transport client.
> 
> On Tue, Jul 19, 2011 at 2:25 AM, lmader [lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) wrote:
> 
> > Could this leak have affected the client as well?
> 
> > Thanks,  
> > Lar
> 
> > On Jul 14, 9:49 am, Mathew Davies [thepixeldevelo...@googlemail.com](mailto:thepixeldevelo...@googlemail.com)  
> > wrote:
> > 
> > > 20 hours have past and I can confirm that Shay has fixed the memory leak.
> 
> > > See my new munin graph here:[http://i.imgur.com/MCGW0.png](http://i.imgur.com/MCGW0.png)
> 
> > > If you can't upgrade your servers to use the latest code from the  
> > > repository  
> > > you can also do what Vertice suggested [1] and pass the
> 
> > > Des.monitor.jvm.enabled=false
> 
> > > flag to the binary.
> 
> > > [1]  
> > > [Memory leak · Issue #1075 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1075#issuecomme)...
> 
> > > Kind Regards,  
> > > -Mathew Davies.
> 
> > > On 13 July 2011 17:14, Mathew Davies \<[thepixeldevelo...@googlemail.com](mailto:thepixeldevelo...@googlemail.com)  
> > > wrote:
> 
> > > > Just to let you know, Kimchy has tracked down and fixed a memory leak.  
> > > > You  
> > > > can see the commit here:
> 
> > > > [Native (java) process memory leak, closes #1118. · elastic/elasticsearch@1033249 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/1033249f0c971ea).  
> > > > ..
> 
> > > > If people can test this out and let us know if it works, that'd be  
> > > > great.
> 
> > > > Kind Regards,  
> > > > -Mathew Davies.
> 
> > > > On 8 July 2011 01:59, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > > Its possible, I am trying to track this down and check.
> 
> > > > > On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> 
> > > > > We are not using sigar directly, unless ES does something with it. We  
> > > > > are running ES as a service.
> 
> > > > > Is it possible anything in the network layer or nio stuff could be  
> > > > > leaking?
> 
> > > > > Thanks so much,  
> > > > > Lar
> 
> > > > > On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > > Thats a difficult one the figure out. elasticsearch does not use  
> > > > > native  
> > > > > memory (by default), and this leak can come from allocations of native  
> > > > > memory, bugs in native libraries used, or leaks in the java process  
> > > > > itself.  
> > > > > Are you using sigar within that app?
> 
> > > > > On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> 
> > > > > I'm not sure I follow your suggestion: As I mentioned, the memory  
> > > > > leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> > > > > monitor shows normal memory usage and garbage collection patterns.  
> > > > > However the java process memory grows until all system memory is gone.
> 
> > > > > On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> 
> > > > > Try to find out which kind ofmemorythe heap is piling up (Perma Old  
> > > > > New).  
> > > > > Tomcat is a notorious PermaGen leaker.
> 
> > > > > -------- Original Message --------  
> > > > > Subject: Re:Memoryproblems  
> > > > > From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre  
> > > > > [....@gmail.com](mailto:....@gmail.com))\>  
> > > > > Date: Wed, July 06, 2011 4:52 pm  
> > > > > To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us ....@  
> > > > > [elasticsearch.com](http://elasticsearch.com))\>  
> > > > > We are seeing the same issue. That is, running elasticsearch server  
> > > > > on linux with the sun jdk, the applicationmemoryseems constant, but  
> > > > > the swapmemorysteadily grows over the course of a week or so, until  
> > > > > we run out of swap space.  
> > > > > Additionally, I believe that our elastic client that runs in a Tomcat  
> > > > > webapp is leaking non-heapmemory. After a period of time, the  
> > > > > processmemory(but not the JVMmemory) starts to grow steadily until  
> > > > > we have to restart the tomcat server. We never get a java "out  
> > > > > ofmemory"  
> > > > > error, instead the java process eventually consumes all of  
> > > > > systemmemory.  
> > > > > We have been doing our best to isolate this, and it really does seem  
> > > > > to be the elastic client that is leaking thememory. Perhaps the  
> > > > > sever side swapmemorygrowth is related.  
> > > > > I think we need help with this one.  
> > > > > Thanks,  
> > > > > Lar- Hide quoted text -
> 
> > > - Show quoted text -- Hide quoted text -
> 
> - Show quoted text -

---

<div class="post-metadata">

**Author:** ![ofavre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofavre/32/116819_2.png) [@ofavre](https://discuss.elastic.co/u/ofavre)\
**Post date:** [July 19, 2011, 1:38pm UTC](https://discuss.elastic.co/t/memory-problems/4661/26 "2011-07-19T13:38:25Z")

</div>

So the fix really consist of doing the reflection magic once and for all,  
instead of doing it at each call?  
I just want to make sure, as I too use such technique in a frequently called  
function.

--  
Olivier Favre

[www.yakaz.com](http://www.yakaz.com)

2011/7/14 Mathew Davies [thepixeldeveloper@googlemail.com](mailto:thepixeldeveloper@googlemail.com)

> 20 hours have past and I can confirm that Shay has fixed the memory leak.
> 
> See my new munin graph here: [http://i.imgur.com/MCGW0.png](http://i.imgur.com/MCGW0.png)
> 
> If you can't upgrade your servers to use the latest code from the  
> repository you can also do what Vertice suggested [1] and pass the
> 
> Des.monitor.jvm.enabled=false
> 
> flag to the binary.
> 
> [1]  
> [Memory leak · Issue #1075 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1075#issuecomment-1571469)
> 
> Kind Regards,  
> -Mathew Davies.
> 
> On 13 July 2011 17:14, Mathew Davies [thepixeldeveloper@googlemail.com](mailto:thepixeldeveloper@googlemail.com)wrote:
> 
> > Just to let you know, Kimchy has tracked down and fixed a memory leak. You  
> > can see the commit here:
> > 
> > [Native (java) process memory leak, closes #1118. · elastic/elasticsearch@1033249 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/1033249f0c971ea099ccbce6d9fe3cd720c7c800)
> > 
> > If people can test this out and let us know if it works, that'd be great.
> > 
> > Kind Regards,  
> > -Mathew Davies.
> > 
> > On 8 July 2011 01:59, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > 
> > > Its possible, I am trying to track this down and check.
> > > 
> > > On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> > > 
> > > We are not using sigar directly, unless ES does something with it. We  
> > > are running ES as a service.
> > > 
> > > Is it possible anything in the network layer or nio stuff could be  
> > > leaking?
> > > 
> > > Thanks so much,  
> > > Lar
> > > 
> > > On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > 
> > > Thats a difficult one the figure out. elasticsearch does not use native  
> > > memory (by default), and this leak can come from allocations of native  
> > > memory, bugs in native libraries used, or leaks in the java process itself.  
> > > Are you using sigar within that app?
> > > 
> > > On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> > > 
> > > I'm not sure I follow your suggestion: As I mentioned, the memory  
> > > leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> > > monitor shows normal memory usage and garbage collection patterns.  
> > > However the java process memory grows until all system memory is gone.
> > > 
> > > On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> > > 
> > > Try to find out which kind ofmemorythe heap is piling up (Perma Old New).  
> > > Tomcat is a notorious PermaGen leaker.
> > > 
> > > -------- Original Message --------  
> > > Subject: Re:Memoryproblems  
> > > From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre   
> > > [...@gmail.com](mailto:...@gmail.com))\>  
> > > Date: Wed, July 06, 2011 4:52 pm  
> > > To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us [...@elasticsearch.com](mailto:...@elasticsearch.com)  
> > > )\>  
> > > We are seeing the same issue. That is, running elasticsearch server  
> > > on linux with the sun jdk, the applicationmemoryseems constant, but  
> > > the swapmemorysteadily grows over the course of a week or so, until  
> > > we run out of swap space.  
> > > Additionally, I believe that our elastic client that runs in a Tomcat  
> > > webapp is leaking non-heapmemory. After a period of time, the  
> > > processmemory(but not the JVMmemory) starts to grow steadily until  
> > > we have to restart the tomcat server. We never get a java "out ofmemory"  
> > > error, instead the java process eventually consumes all of  
> > > systemmemory.  
> > > We have been doing our best to isolate this, and it really does seem  
> > > to be the elastic client that is leaking thememory. Perhaps the  
> > > sever side swapmemorygrowth is related.  
> > > I think we need help with this one.  
> > > Thanks,  
> > > Lar

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 19, 2011, 9:17pm UTC](https://discuss.elastic.co/t/memory-problems/4661/27 "2011-07-19T21:17:35Z")

</div>

No, it has nothing to do with reflection. Its simply that calling  
getLastGCInfo method on the GC MX bean leaks native memory every call.

On Tue, Jul 19, 2011 at 4:38 PM, Olivier Favre [olivier@yakaz.com](mailto:olivier@yakaz.com) wrote:

> So the fix really consist of doing the reflection magic once and for all,  
> instead of doing it at each call?  
> I just want to make sure, as I too use such technique in a frequently  
> called function.
> 
> --  
> Olivier Favre
> 
> [www.yakaz.com](http://www.yakaz.com)
> 
> 2011/7/14 Mathew Davies [thepixeldeveloper@googlemail.com](mailto:thepixeldeveloper@googlemail.com)
> 
> > 20 hours have past and I can confirm that Shay has fixed the memory leak.
> > 
> > See my new munin graph here: [http://i.imgur.com/MCGW0.png](http://i.imgur.com/MCGW0.png)
> > 
> > If you can't upgrade your servers to use the latest code from the  
> > repository you can also do what Vertice suggested [1] and pass the
> > 
> > Des.monitor.jvm.enabled=false
> > 
> > flag to the binary.
> > 
> > [1]  
> > [Memory leak · Issue #1075 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1075#issuecomment-1571469)
> > 
> > Kind Regards,  
> > -Mathew Davies.
> > 
> > On 13 July 2011 17:14, Mathew Davies [thepixeldeveloper@googlemail.com](mailto:thepixeldeveloper@googlemail.com)wrote:
> > 
> > > Just to let you know, Kimchy has tracked down and fixed a memory leak.  
> > > You can see the commit here:
> > > 
> > > [Native (java) process memory leak, closes #1118. · elastic/elasticsearch@1033249 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/1033249f0c971ea099ccbce6d9fe3cd720c7c800)
> > > 
> > > If people can test this out and let us know if it works, that'd be great.
> > > 
> > > Kind Regards,  
> > > -Mathew Davies.
> > > 
> > > On 8 July 2011 01:59, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > > 
> > > > Its possible, I am trying to track this down and check.
> > > > 
> > > > On Friday, July 8, 2011 at 12:28 AM, lmader wrote:
> > > > 
> > > > We are not using sigar directly, unless ES does something with it. We  
> > > > are running ES as a service.
> > > > 
> > > > Is it possible anything in the network layer or nio stuff could be  
> > > > leaking?
> > > > 
> > > > Thanks so much,  
> > > > Lar
> > > > 
> > > > On Jul 7, 7:20 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > > 
> > > > Thats a difficult one the figure out. elasticsearch does not use native  
> > > > memory (by default), and this leak can come from allocations of native  
> > > > memory, bugs in native libraries used, or leaks in the java process itself.  
> > > > Are you using sigar within that app?
> > > > 
> > > > On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:
> > > > 
> > > > I'm not sure I follow your suggestion: As I mentioned, the memory  
> > > > leak is _not_ in the JVM memory - _not_ in the regular heaps. The JVM  
> > > > monitor shows normal memory usage and garbage collection patterns.  
> > > > However the java process memory grows until all system memory is gone.
> > > > 
> > > > On Jul 6, 3:26 pm, \<[jp.lora...@cfyar.com](mailto:jp.lora...@cfyar.com) ([http://cfyar.com](http://cfyar.com))\> wrote:
> > > > 
> > > > Try to find out which kind ofmemorythe heap is piling up (Perma Old  
> > > > New). Tomcat is a notorious PermaGen leaker.
> > > > 
> > > > -------- Original Message --------  
> > > > Subject: Re:Memoryproblems  
> > > > From: lmader \<[lmaderintre...@gmail.com](mailto:lmaderintre...@gmail.com) (mailto:lmaderintre  
> > > > [...@gmail.com](mailto:...@gmail.com))\>  
> > > > Date: Wed, July 06, 2011 4:52 pm  
> > > > To: users \<[us...@elasticsearch.com](mailto:us...@elasticsearch.com) (mailto:us [...@elasticsearch.com](mailto:...@elasticsearch.com)  
> > > > )\>  
> > > > We are seeing the same issue. That is, running elasticsearch server  
> > > > on linux with the sun jdk, the applicationmemoryseems constant, but  
> > > > the swapmemorysteadily grows over the course of a week or so, until  
> > > > we run out of swap space.  
> > > > Additionally, I believe that our elastic client that runs in a Tomcat  
> > > > webapp is leaking non-heapmemory. After a period of time, the  
> > > > processmemory(but not the JVMmemory) starts to grow steadily until  
> > > > we have to restart the tomcat server. We never get a java "out ofmemory"  
> > > > error, instead the java process eventually consumes all of  
> > > > systemmemory.  
> > > > We have been doing our best to isolate this, and it really does seem  
> > > > to be the elastic client that is leaking thememory. Perhaps the  
> > > > sever side swapmemorygrowth is related.  
> > > > I think we need help with this one.  
> > > > Thanks,  
> > > > Lar

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:00am UTC](https://discuss.elastic.co/t/memory-problems/4661/28 "2017-07-06T04:00:08Z")

</div>



[Previous page](https://discuss.elastic.co/t/memory-problems/4661.md?page=1)
