Memory Leak?

Elastic Stack Elasticsearch
1

Hi,

I ran a load test on a single node (with a default configuration) and
found that there might be a memory leak in the SoftFilterCache.

The test used 20 concurrent users that added a document once every ~5
seconds.
10 additional concurrent users simulated searching every ~1 minute.

After about 12 hours the JVM heap ran out of space (1G) and
ElasticSearch went into repeating GC cycles that caused it to stop
responding.

The heap dump shows that the problematic file is
org.elasticsearch.index.cache.filter.soft.SoftFilterCache ->
org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap

holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which
occupied about 660M.

Heap memory usage graph: http://twitpic.com/1mm50a

Is there any configuration item I'm missing?

Thanks,
Tal

2

Hi,

The leak is not really there, its just that the cache evicts when readers
are not used anymore. I think I have fixed this problem in master, can you
give it a go?

cheers,
shay.banon

On Mon, May 10, 2010 at 7:49 AM, Tal talsalmona@gmail.com wrote:

Hi,

I ran a load test on a single node (with a default configuration) and
found that there might be a memory leak in the SoftFilterCache.

The test used 20 concurrent users that added a document once every ~5
seconds.
10 additional concurrent users simulated searching every ~1 minute.

After about 12 hours the JVM heap ran out of space (1G) and
Elasticsearch went into repeating GC cycles that caused it to stop
responding.

The heap dump shows that the problematic file is
org.elasticsearch.index.cache.filter.soft.SoftFilterCache ->
org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap

holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which
occupied about 660M.

Heap memory usage graph: http://twitpic.com/1mm50a

Is there any configuration item I'm missing?

Thanks,
Tal

3

Will give it a try and keep you posted.
Thanks for the great work.

Tal

On May 10, 9:45 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

The leak is not really there, its just that the cache evicts when readers
are not used anymore. I think I have fixed this problem in master, can you
give it a go?

cheers,
shay.banon

On Mon, May 10, 2010 at 7:49 AM, Tal talsalm...@gmail.com wrote:

Hi,

I ran a load test on a single node (with a default configuration) and
found that there might be a memory leak in the SoftFilterCache.

The test used 20 concurrent users that added a document once every ~5
seconds.
10 additional concurrent users simulated searching every ~1 minute.

After about 12 hours the JVM heap ran out of space (1G) and
Elasticsearch went into repeating GC cycles that caused it to stop
responding.

The heap dump shows that the problematic file is
org.elasticsearch.index.cache.filter.soft.SoftFilterCache ->
org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap

holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which
occupied about 660M.

Heap memory usage graph:http://twitpic.com/1mm50a

Is there any configuration item I'm missing?

Thanks,
Tal

4

Hi Shay,

After running a more significant load test on Elasticsearch-0.7.0-
SNAPSHOT I see that the memory leak is still there.
More details here: http://docs.google.com/View?id=dd9m74r8_86dtf48wcj

My store configuration is:
index :
store:
fs:
memory:
enabled: true

I will try to do another test where the memory.enabled=false to see
how it affects the results.
In the mean time, do you have any suggestions on how this can be
resolved?

Thanks,
Tal

On May 10, 8:38 pm, Tal talsalm...@gmail.com wrote:

Will give it a try and keep you posted.
Thanks for the great work.

Tal

On May 10, 9:45 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

The leak is not really there, its just that the cache evicts when readers
are not used anymore. I think I have fixed this problem in master, can you
give it a go?

cheers,
shay.banon

On Mon, May 10, 2010 at 7:49 AM, Tal talsalm...@gmail.com wrote:

Hi,

I ran a load test on a single node (with a default configuration) and
found that there might be a memory leak in the SoftFilterCache.

The test used 20 concurrent users that added a document once every ~5
seconds.
10 additional concurrent users simulated searching every ~1 minute.

After about 12 hours the JVM heap ran out of space (1G) and
Elasticsearch went into repeating GC cycles that caused it to stop
responding.

The heap dump shows that the problematic file is
org.elasticsearch.index.cache.filter.soft.SoftFilterCache ->
org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap

holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which
occupied about 660M.

Heap memory usage graph:ElasticSearch memory leak?

Is there any configuration item I'm missing?

Thanks,
Tal

5

Is there a chance that I can run this test? Can you send it to me with
details on how to run it?

On Fri, May 14, 2010 at 12:50 PM, Tal talsalmona@gmail.com wrote:

Hi Shay,

After running a more significant load test on Elasticsearch-0.7.0-
SNAPSHOT I see that the memory leak is still there.
More details here: http://docs.google.com/View?id=dd9m74r8_86dtf48wcj

My store configuration is:
index :
store:
fs:
memory:
enabled: true

I will try to do another test where the memory.enabled=false to see
how it affects the results.
In the mean time, do you have any suggestions on how this can be
resolved?

Thanks,
Tal

On May 10, 8:38 pm, Tal talsalm...@gmail.com wrote:

Will give it a try and keep you posted.
Thanks for the great work.

Tal

On May 10, 9:45 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

The leak is not really there, its just that the cache evicts when
readers
are not used anymore. I think I have fixed this problem in master, can
you
give it a go?

cheers,
shay.banon

On Mon, May 10, 2010 at 7:49 AM, Tal talsalm...@gmail.com wrote:

Hi,

I ran a load test on a single node (with a default configuration) and
found that there might be a memory leak in the SoftFilterCache.

The test used 20 concurrent users that added a document once every ~5
seconds.
10 additional concurrent users simulated searching every ~1 minute.

After about 12 hours the JVM heap ran out of space (1G) and
Elasticsearch went into repeating GC cycles that caused it to stop
responding.

The heap dump shows that the problematic file is
org.elasticsearch.index.cache.filter.soft.SoftFilterCache ->
org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap

holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders which
occupied about 660M.

Heap memory usage graph:http://twitpic.com/1mm50a

Is there any configuration item I'm missing?

Thanks,
Tal

6

I just ran some stress tests before the upcoming 0.7 release and I don't see
what you are getting (ran 5 billion index / search requests on a 256m single
node JVM and it seems to be good - single index). So, I would need to know
exactly what you do against elasticsearch to be able to recreate this... .

By the way, if you know jmeter, I have several benchmark scripts that I use
(its in the source repo), if you can simulate it using a custom script, it
would be great.

cheers,
shay.banon

On Fri, May 14, 2010 at 12:55 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Is there a chance that I can run this test? Can you send it to me with
details on how to run it?

On Fri, May 14, 2010 at 12:50 PM, Tal talsalmona@gmail.com wrote:

Hi Shay,

After running a more significant load test on Elasticsearch-0.7.0-
SNAPSHOT I see that the memory leak is still there.
More details here: http://docs.google.com/View?id=dd9m74r8_86dtf48wcj

My store configuration is:
index :
store:
fs:
memory:
enabled: true

I will try to do another test where the memory.enabled=false to see
how it affects the results.
In the mean time, do you have any suggestions on how this can be
resolved?

Thanks,
Tal

On May 10, 8:38 pm, Tal talsalm...@gmail.com wrote:

Will give it a try and keep you posted.
Thanks for the great work.

Tal

On May 10, 9:45 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

The leak is not really there, its just that the cache evicts when
readers
are not used anymore. I think I have fixed this problem in master, can
you
give it a go?

cheers,
shay.banon

On Mon, May 10, 2010 at 7:49 AM, Tal talsalm...@gmail.com wrote:

Hi,

I ran a load test on a single node (with a default configuration)
and
found that there might be a memory leak in the SoftFilterCache.

The test used 20 concurrent users that added a document once every
~5
seconds.
10 additional concurrent users simulated searching every ~1 minute.

After about 12 hours the JVM heap ran out of space (1G) and
Elasticsearch went into repeating GC cycles that caused it to stop
responding.

The heap dump shows that the problematic file is
org.elasticsearch.index.cache.filter.soft.SoftFilterCache ->
org.elasticsearch.util.concurrent.highscalelib.NonBlockingHashMap

holding 26600 org.apache.lucene.index.SegmentReader$CoreReaders
which
occupied about 660M.

Heap memory usage graph:ElasticSearch memory leak?

Is there any configuration item I'm missing?

Thanks,
Tal