# Memory Leak? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 10, 2010, 4:49am UTC](https://discuss.elastic.co/t/memory-leak/2944 "2010-05-10T04:49:19Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\ **Post date:** [May 10, 2010, 4:49am UTC](https://discuss.elastic.co/t/memory-leak/2944/1 "2010-05-10T04:49:19Z")
Hi, I ran a load test on a single node (with a default configuration) and found that there might be a memory leak in the SoftFilterCache. The test used 20 concurrent users that added a document once every ~5 seconds. 10 additional concurrent users simulated searching every ~1 minute. After about 12 hours the JVM heap ran out of space (1G) and ElasticSearch went into repeating GC cycles that caused it to stop responding. The heap dump shows that the problematic file is org.elasticsearch.index.cache.filter.soft.SoftFilterCache -\> org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which occupied about 660M. Heap memory usage graph: [http://twitpic.com/1mm50a](http://twitpic.com/1mm50a) Is there any configuration item I'm missing? Thanks, Tal ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 10, 2010, 6:45am UTC](https://discuss.elastic.co/t/memory-leak/2944/2 "2010-05-10T06:45:37Z")
Hi, The leak is not really there, its just that the cache evicts when readers are not used anymore. I think I have fixed this problem in master, can you give it a go? cheers, shay.banon On Mon, May 10, 2010 at 7:49 AM, Tal [talsalmona@gmail.com](mailto:talsalmona@gmail.com) wrote: > Hi, > > I ran a load test on a single node (with a default configuration) and > found that there might be a memory leak in the SoftFilterCache. > > The test used 20 concurrent users that added a document once every ~5 > seconds. > 10 additional concurrent users simulated searching every ~1 minute. > > After about 12 hours the JVM heap ran out of space (1G) and > Elasticsearch went into repeating GC cycles that caused it to stop > responding. > > The heap dump shows that the problematic file is > org.elasticsearch.index.cache.filter.soft.SoftFilterCache -\> > org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap > > holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which > occupied about 660M. > > Heap memory usage graph: [http://twitpic.com/1mm50a](http://twitpic.com/1mm50a) > > Is there any configuration item I'm missing? > > Thanks, > Tal ---
**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\ **Post date:** [May 10, 2010, 5:38pm UTC](https://discuss.elastic.co/t/memory-leak/2944/3 "2010-05-10T17:38:21Z")
Will give it a try and keep you posted. Thanks for the great work. Tal On May 10, 9:45 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > The leak is not really there, its just that the cache evicts when readers > are not used anymore. I think I have fixed this problem in master, can you > give it a go? > > cheers, > shay.banon > > On Mon, May 10, 2010 at 7:49 AM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > Hi, > > > I ran a load test on a single node (with a default configuration) and > > found that there might be a memory leak in the SoftFilterCache. > > > The test used 20 concurrent users that added a document once every ~5 > > seconds. > > 10 additional concurrent users simulated searching every ~1 minute. > > > After about 12 hours the JVM heap ran out of space (1G) and > > Elasticsearch went into repeating GC cycles that caused it to stop > > responding. > > > The heap dump shows that the problematic file is > > org.elasticsearch.index.cache.filter.soft.SoftFilterCache -\> > > org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap > > > holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which > > occupied about 660M. > > > Heap memory usage graph:[http://twitpic.com/1mm50a](http://twitpic.com/1mm50a) > > > Is there any configuration item I'm missing? > > > Thanks, > > Tal ---
**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\ **Post date:** [May 14, 2010, 9:50am UTC](https://discuss.elastic.co/t/memory-leak/2944/4 "2010-05-14T09:50:42Z")
Hi Shay, After running a more significant load test on Elasticsearch-0.7.0- SNAPSHOT I see that the memory leak is still there. More details here: [http://docs.google.com/View?id=dd9m74r8\_86dtf48wcj](http://docs.google.com/View?id=dd9m74r8_86dtf48wcj) My store configuration is: index : store: fs: memory: enabled: true I will try to do another test where the memory.enabled=false to see how it affects the results. In the mean time, do you have any suggestions on how this can be resolved? Thanks, Tal On May 10, 8:38 pm, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > Will give it a try and keep you posted. > Thanks for the great work. > > Tal > > On May 10, 9:45 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi, > > > The leak is not really there, its just that the cache evicts when readers > > are not used anymore. I think I have fixed this problem in master, can you > > give it a go? > > > cheers, > > shay.banon > > > On Mon, May 10, 2010 at 7:49 AM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > > > Hi, > > > > I ran a load test on a single node (with a default configuration) and > > > found that there might be a memory leak in the SoftFilterCache. > > > > The test used 20 concurrent users that added a document once every ~5 > > > seconds. > > > 10 additional concurrent users simulated searching every ~1 minute. > > > > After about 12 hours the JVM heap ran out of space (1G) and > > > Elasticsearch went into repeating GC cycles that caused it to stop > > > responding. > > > > The heap dump shows that the problematic file is > > > org.elasticsearch.index.cache.filter.soft.SoftFilterCache -\> > > > org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap > > > > holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which > > > occupied about 660M. > > > > Heap memory usage graph:[ElasticSearch memory leak?](http://twitpic.com/1mm50a) > > > > Is there any configuration item I'm missing? > > > > Thanks, > > > Tal ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 14, 2010, 9:55am UTC](https://discuss.elastic.co/t/memory-leak/2944/5 "2010-05-14T09:55:20Z")
Is there a chance that I can run this test? Can you send it to me with details on how to run it? On Fri, May 14, 2010 at 12:50 PM, Tal [talsalmona@gmail.com](mailto:talsalmona@gmail.com) wrote: > Hi Shay, > > After running a more significant load test on Elasticsearch-0.7.0- > SNAPSHOT I see that the memory leak is still there. > More details here: [http://docs.google.com/View?id=dd9m74r8\_86dtf48wcj](http://docs.google.com/View?id=dd9m74r8_86dtf48wcj) > > My store configuration is: > index : > store: > fs: > memory: > enabled: true > > I will try to do another test where the memory.enabled=false to see > how it affects the results. > In the mean time, do you have any suggestions on how this can be > resolved? > > Thanks, > Tal > > On May 10, 8:38 pm, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > Will give it a try and keep you posted. > > Thanks for the great work. > > > > Tal > > > > On May 10, 9:45 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hi, > > > > > The leak is not really there, its just that the cache evicts when > > > readers > > > are not used anymore. I think I have fixed this problem in master, can > > > you > > > give it a go? > > > > > cheers, > > > shay.banon > > > > > On Mon, May 10, 2010 at 7:49 AM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > > > > > Hi, > > > > > > I ran a load test on a single node (with a default configuration) and > > > > found that there might be a memory leak in the SoftFilterCache. > > > > > > The test used 20 concurrent users that added a document once every ~5 > > > > seconds. > > > > 10 additional concurrent users simulated searching every ~1 minute. > > > > > > After about 12 hours the JVM heap ran out of space (1G) and > > > > Elasticsearch went into repeating GC cycles that caused it to stop > > > > responding. > > > > > > The heap dump shows that the problematic file is > > > > org.elasticsearch.index.cache.filter.soft.SoftFilterCache -\> > > > > org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap > > > > > > holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which > > > > occupied about 660M. > > > > > > Heap memory usage graph:[http://twitpic.com/1mm50a](http://twitpic.com/1mm50a) > > > > > > Is there any configuration item I'm missing? > > > > > > Thanks, > > > > Tal ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 14, 2010, 2:08pm UTC](https://discuss.elastic.co/t/memory-leak/2944/6 "2010-05-14T14:08:23Z")
I just ran some stress tests before the upcoming 0.7 release and I don't see what you are getting (ran 5 billion index / search requests on a 256m single node JVM and it seems to be good - single index). So, I would need to know exactly what you do against elasticsearch to be able to recreate this... . By the way, if you know jmeter, I have several benchmark scripts that I use (its in the source repo), if you can simulate it using a custom script, it would be great. cheers, shay.banon On Fri, May 14, 2010 at 12:55 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Is there a chance that I can run this test? Can you send it to me with > details on how to run it? > > On Fri, May 14, 2010 at 12:50 PM, Tal [talsalmona@gmail.com](mailto:talsalmona@gmail.com) wrote: > > > Hi Shay, > > > > After running a more significant load test on Elasticsearch-0.7.0- > > SNAPSHOT I see that the memory leak is still there. > > More details here: [http://docs.google.com/View?id=dd9m74r8\_86dtf48wcj](http://docs.google.com/View?id=dd9m74r8_86dtf48wcj) > > > > My store configuration is: > > index : > > store: > > fs: > > memory: > > enabled: true > > > > I will try to do another test where the memory.enabled=false to see > > how it affects the results. > > In the mean time, do you have any suggestions on how this can be > > resolved? > > > > Thanks, > > Tal > > > > On May 10, 8:38 pm, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > > > Will give it a try and keep you posted. > > > Thanks for the great work. > > > > > > Tal > > > > > > On May 10, 9:45 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Hi, > > > > > > > The leak is not really there, its just that the cache evicts when > > > > readers > > > > are not used anymore. I think I have fixed this problem in master, can > > > > you > > > > give it a go? > > > > > > > cheers, > > > > shay.banon > > > > > > > On Mon, May 10, 2010 at 7:49 AM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > I ran a load test on a single node (with a default configuration) > > > > > and > > > > > found that there might be a memory leak in the SoftFilterCache. > > > > > > > > The test used 20 concurrent users that added a document once every > > > > > ~5 > > > > > seconds. > > > > > 10 additional concurrent users simulated searching every ~1 minute. > > > > > > > > After about 12 hours the JVM heap ran out of space (1G) and > > > > > Elasticsearch went into repeating GC cycles that caused it to stop > > > > > responding. > > > > > > > > The heap dump shows that the problematic file is > > > > > org.elasticsearch.index.cache.filter.soft.SoftFilterCache -\> > > > > > org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap > > > > > > > > holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders > > > > > which > > > > > occupied about 660M. > > > > > > > > Heap memory usage graph:[ElasticSearch memory leak?](http://twitpic.com/1mm50a) > > > > > > > > Is there any configuration item I'm missing? > > > > > > > > Thanks, > > > > > Tal ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:24am UTC](https://discuss.elastic.co/t/memory-leak/2944/7 "2017-07-06T04:24:09Z")