ElasticSearch OutOfMemory Exceptions

Michel_Conrad · September 23, 2011, 1:31pm

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap
takes 3.8 GB.

gist.github.com

https://gist.github.com/anonymous/1237322

gistfile1.txt

Class Name                                                                                                      | Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------------------------
org.elasticsearch.common.concurrentlinkedhashmap.ConcurrentLinkedHashMap @ 0x685762218                          |          160 | 3,890,859,000
|- <class> class org.elasticsearch.common.concurrentlinkedhashmap.ConcurrentLinkedHashMap @ 0x7fc17d340         |           48 |            64
|- executor java.util.concurrent.ScheduledThreadPoolExecutor @ 0x68567f298                                      |          112 |        10,024
|- listener org.elasticsearch.indices.cache.filter.IndicesNodeFilterCache @ 0x685761f80                         |           64 |           344
|- data java.util.concurrent.ConcurrentHashMap @ 0x685762278                                                    |           72 |   100,281,088
|- evictionLock java.util.concurrent.locks.ReentrantLock @ 0x685762778                                          |           24 |            72
|- evictionDeque org.elasticsearch.common.concurrentlinkedhashmap.LinkedDeque @ 0x6857627b0                     |           40 |            40
|- weigher org.elasticsearch.common.concurrentlinkedhashmap.ConcurrentLinkedHashMap$BoundedWeigher @ 0x6857627c8|           24 |            40

This file has been truncated. show original

Is there a possibility to log the queries the individual nodes handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

kimchy · September 23, 2011, 11:12pm

How much memory do you allocate to the instance? This memory reference to
the filter cache, are you maybe using large terms filters with a lot of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad <
michel.conrad@trendiction.com> wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

Michel_Conrad · September 26, 2011, 7:52am

Hi Shay,
I am allocating 6G to the ES instance and am using quite a lot of
terms filters. A single
call to a filter should although not have too many terms (30
non-analyzed urls).

Is there a way to limit the size the cache can take? By reading the
documentation I thought that
the cache would by default max to 20% of the heap size, but in my case
the cache is at 3.8GB
if I understand it correctly.

Best Regards,
Michel

On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon kimchy@gmail.com wrote:

How much memory do you allocate to the instance? This memory reference to
the filter cache, are you maybe using large terms filters with a lot of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

kimchy · September 26, 2011, 8:28am

Hi,

It might be related to the filters you use then. The filters cache has
two aspects to it, the cache key and the values. The memory limit on it is
computed against the values. Its quite difficult to compute the memory used
for the keys (which are the filters themselves), but, they obviously would
also use memory. For this reason, there is the possibility to use a custom
_cache_key on all the filters (including the terms filter) where you can
control the key under which the filter will be used. Can you use that in
someway?

When we added the _cache_key option, one other option was to have an
option and use something like md5 on the terms filters values to
automatically generate a cache key, but its not in yet...

-shay.banon

On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad <
michel.conrad@trendiction.com> wrote:

Hi Shay,
I am allocating 6G to the ES instance and am using quite a lot of
terms filters. A single
call to a filter should although not have too many terms (30
non-analyzed urls).

Is there a way to limit the size the cache can take? By reading the
documentation I thought that
the cache would by default max to 20% of the heap size, but in my case
the cache is at 3.8GB
if I understand it correctly.

Best Regards,
Michel

On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon kimchy@gmail.com wrote:

How much memory do you allocate to the instance? This memory reference to
the filter cache, are you maybe using large terms filters with a lot of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

Michel_Conrad · September 26, 2011, 9:06am

Hi Shay,

I had another look at the heap dump regarding your comments,
and for me it looks as if the FilterCacheValues take up over 3GB,
therefore I still don't understand why the cache gets so big.

I pasted a dominator tree of the filtercache grouped by class.

gist.github.com

https://gist.github.com/anonymous/eaaa3df2e5c3f8f62dc1

gistfile1.txt

Class Name                                                                                  |   Objects | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------------------
                                                                                            |           |              |               |           
org.elasticsearch.common.concurrentlinkedhashmap.ConcurrentLinkedHashMap                    |         1 |          160 | 3,890,859,000 |     56.80%
|- org.elasticsearch.common.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node            | 1,111,546 |   62,246,576 | 3,481,146,984 |     50.82%
|  '- org.elasticsearch.common.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue| 1,111,546 |   35,569,472 | 3,418,900,408 |     49.91%
|     '- org.elasticsearch.index.cache.filter.support.FilterCacheValue                      | 1,111,546 |   35,569,472 | 3,383,330,936 |     49.39%
|        '- org.elasticsearch.common.lucene.docset.OpenBitDocSet                            | 1,104,048 |   26,497,152 | 3,347,761,464 |     48.87%
|           |- org.apache.lucene.util.OpenBitSet                                            |   641,370 |   25,654,800 | 1,916,982,288 |     27.98%
|           |- org.apache.lucene.util.OpenBitSetDISI                                        |   462,678 |   18,507,120 | 1,404,282,024 |     20.50%

This file has been truncated. show original

Best,
Michel

On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon kimchy@gmail.com wrote:

Hi,
It might be related to the filters you use then. The filters cache has
two aspects to it, the cache key and the values. The memory limit on it is
computed against the values. Its quite difficult to compute the memory used
for the keys (which are the filters themselves), but, they obviously would
also use memory. For this reason, there is the possibility to use a custom
_cache_key on all the filters (including the terms filter) where you can
control the key under which the filter will be used. Can you use that in
someway?
When we added the _cache_key option, one other option was to have an
option and use something like md5 on the terms filters values to
automatically generate a cache key, but its not in yet...
-shay.banon

On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi Shay,
I am allocating 6G to the ES instance and am using quite a lot of
terms filters. A single
call to a filter should although not have too many terms (30
non-analyzed urls).

Is there a way to limit the size the cache can take? By reading the
documentation I thought that
the cache would by default max to 20% of the heap size, but in my case
the cache is at 3.8GB
if I understand it correctly.

Best Regards,
Michel

On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon kimchy@gmail.com wrote:

How much memory do you allocate to the instance? This memory reference
to
the filter cache, are you maybe using large terms filters with a lot of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

kimchy · September 26, 2011, 9:35am

What are you using to navigate the heap dump? Is it the stats for live
objects only? I ran a quick test and I can see the 20% threshold nicely
maintained even when profiling (putting aside cache keys, as I explained
before).

On Mon, Sep 26, 2011 at 12:06 PM, Michel Conrad <
michel.conrad@trendiction.com> wrote:

Hi Shay,

I had another look at the heap dump regarding your comments,
and for me it looks as if the FilterCacheValues take up over 3GB,
therefore I still don't understand why the cache gets so big.

I pasted a dominator tree of the filtercache grouped by class.
gist:eaaa3df2e5c3f8f62dc1 · GitHub

Best,
Michel

On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon kimchy@gmail.com wrote:

Hi,
It might be related to the filters you use then. The filters cache has
two aspects to it, the cache key and the values. The memory limit on it
is
computed against the values. Its quite difficult to compute the memory
used
for the keys (which are the filters themselves), but, they obviously
would
also use memory. For this reason, there is the possibility to use a
custom
_cache_key on all the filters (including the terms filter) where you can
control the key under which the filter will be used. Can you use that in
someway?
When we added the _cache_key option, one other option was to have an
option and use something like md5 on the terms filters values to
automatically generate a cache key, but its not in yet...
-shay.banon

On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi Shay,
I am allocating 6G to the ES instance and am using quite a lot of
terms filters. A single
call to a filter should although not have too many terms (30
non-analyzed urls).

Is there a way to limit the size the cache can take? By reading the
documentation I thought that
the cache would by default max to 20% of the heap size, but in my case
the cache is at 3.8GB
if I understand it correctly.

Best Regards,
Michel

On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon kimchy@gmail.com wrote:

How much memory do you allocate to the instance? This memory reference
to
the filter cache, are you maybe using large terms filters with a lot
of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes
handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

Michel_Conrad · September 26, 2011, 11:16am

Hi,
I'm using the Eclipse Memory Analyzer.
I click on the ConcurrentLinkedHashMap / Java Basics / Open in
dominator tree with grouping the objects by class.
This is where I get over 3Gb retained heap for FilterCacheValues from.

Best,
Michel

On Mon, Sep 26, 2011 at 11:35 AM, Shay Banon kimchy@gmail.com wrote:

What are you using to navigate the heap dump? Is it the stats for live
objects only? I ran a quick test and I can see the 20% threshold nicely
maintained even when profiling (putting aside cache keys, as I explained
before).

On Mon, Sep 26, 2011 at 12:06 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi Shay,

I had another look at the heap dump regarding your comments,
and for me it looks as if the FilterCacheValues take up over 3GB,
therefore I still don't understand why the cache gets so big.

I pasted a dominator tree of the filtercache grouped by class.
gist:eaaa3df2e5c3f8f62dc1 · GitHub

Best,
Michel

On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon kimchy@gmail.com wrote:

Hi,
It might be related to the filters you use then. The filters cache
has
two aspects to it, the cache key and the values. The memory limit on it
is
computed against the values. Its quite difficult to compute the memory
used
for the keys (which are the filters themselves), but, they obviously
would
also use memory. For this reason, there is the possibility to use a
custom
_cache_key on all the filters (including the terms filter) where you can
control the key under which the filter will be used. Can you use that in
someway?
When we added the _cache_key option, one other option was to have an
option and use something like md5 on the terms filters values to
automatically generate a cache key, but its not in yet...
-shay.banon

On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi Shay,
I am allocating 6G to the ES instance and am using quite a lot of
terms filters. A single
call to a filter should although not have too many terms (30
non-analyzed urls).

Is there a way to limit the size the cache can take? By reading the
documentation I thought that
the cache would by default max to 20% of the heap size, but in my case
the cache is at 3.8GB
if I understand it correctly.

Best Regards,
Michel

On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon kimchy@gmail.com wrote:

How much memory do you allocate to the instance? This memory
reference
to
the filter cache, are you maybe using large terms filters with a lot
of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single
ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes
handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel

kimchy · September 26, 2011, 3:59pm

Haven't used EMA for a long time, so I don't know if what you do excludes
uncollected references.

On Mon, Sep 26, 2011 at 2:16 PM, Michel Conrad <
michel.conrad@trendiction.com> wrote:

Hi,
I'm using the Eclipse Memory Analyzer.
I click on the ConcurrentLinkedHashMap / Java Basics / Open in
dominator tree with grouping the objects by class.
This is where I get over 3Gb retained heap for FilterCacheValues from.

Best,
Michel

On Mon, Sep 26, 2011 at 11:35 AM, Shay Banon kimchy@gmail.com wrote:

What are you using to navigate the heap dump? Is it the stats for live
objects only? I ran a quick test and I can see the 20% threshold nicely
maintained even when profiling (putting aside cache keys, as I explained
before).

On Mon, Sep 26, 2011 at 12:06 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi Shay,

I had another look at the heap dump regarding your comments,
and for me it looks as if the FilterCacheValues take up over 3GB,
therefore I still don't understand why the cache gets so big.

I pasted a dominator tree of the filtercache grouped by class.
gist:eaaa3df2e5c3f8f62dc1 · GitHub

Best,
Michel

On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon kimchy@gmail.com wrote:

Hi,
It might be related to the filters you use then. The filters cache
has
two aspects to it, the cache key and the values. The memory limit on
it
is
computed against the values. Its quite difficult to compute the memory
used
for the keys (which are the filters themselves), but, they obviously
would
also use memory. For this reason, there is the possibility to use a
custom
_cache_key on all the filters (including the terms filter) where you
can
control the key under which the filter will be used. Can you use that
in
someway?
When we added the _cache_key option, one other option was to have an
option and use something like md5 on the terms filters values to
automatically generate a cache key, but its not in yet...
-shay.banon

On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi Shay,
I am allocating 6G to the ES instance and am using quite a lot of
terms filters. A single
call to a filter should although not have too many terms (30
non-analyzed urls).

Is there a way to limit the size the cache can take? By reading the
documentation I thought that
the cache would by default max to 20% of the heap size, but in my
case
the cache is at 3.8GB
if I understand it correctly.

Best Regards,
Michel

On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon kimchy@gmail.com
wrote:

How much memory do you allocate to the instance? This memory
reference
to
the filter cache, are you maybe using large terms filters with a
lot
of
values?

On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad
michel.conrad@trendiction.com wrote:

Hi,
lately I am getting OOM Exceptions in ES and I suspect that it
might
have something to do with requests I'm sending to the ES cluster.

I pasted part of the heapdump I got, a single
ConcurrentLinkedHashMap
takes 3.8 GB.
gist:1237322 · GitHub

Is there a possibility to log the queries the individual nodes
handle,
so I might see what happens immediately before the OOM?

Thanks,
Michel