# ElasticSearch OutOfMemory Exceptions **URL:** \ **Category:** Elasticsearch\ **Created:** [September 23, 2011, 1:31pm UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441 "2011-09-23T13:31:37Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [September 23, 2011, 1:31pm UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/1 "2011-09-23T13:31:37Z")
Hi, lately I am getting OOM Exceptions in ES and I suspect that it might have something to do with requests I'm sending to the ES cluster. I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap takes 3.8 GB. > Is there a possibility to log the queries the individual nodes handle, so I might see what happens immediately before the OOM? Thanks, Michel ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 23, 2011, 11:12pm UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/2 "2011-09-23T23:12:48Z")
How much memory do you allocate to the instance? This memory reference to the filter cache, are you maybe using large terms filters with a lot of values? On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad \< [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com)\> wrote: > Hi, > lately I am getting OOM Exceptions in ES and I suspect that it might > have something to do with requests I'm sending to the ES cluster. > > I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap > takes 3.8 GB. > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > Is there a possibility to log the queries the individual nodes handle, > so I might see what happens immediately before the OOM? > > Thanks, > Michel ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [September 26, 2011, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/3 "2011-09-26T07:52:17Z")
Hi Shay, I am allocating 6G to the ES instance and am using quite a lot of terms filters. A single call to a filter should although not have too many terms (30 non-analyzed urls). Is there a way to limit the size the cache can take? By reading the documentation I thought that the cache would by default max to 20% of the heap size, but in my case the cache is at 3.8GB if I understand it correctly. Best Regards, Michel On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > How much memory do you allocate to the instance? This memory reference to > the filter cache, are you maybe using large terms filters with a lot of > values? > > On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > Hi, > > lately I am getting OOM Exceptions in ES and I suspect that it might > > have something to do with requests I'm sending to the ES cluster. > > > > I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap > > takes 3.8 GB. > > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > > > Is there a possibility to log the queries the individual nodes handle, > > so I might see what happens immediately before the OOM? > > > > Thanks, > > Michel ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 26, 2011, 8:28am UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/4 "2011-09-26T08:28:37Z")
Hi, It might be related to the filters you use then. The filters cache has two aspects to it, the cache key and the values. The memory limit on it is computed against the values. Its quite difficult to compute the memory used for the keys (which are the filters themselves), but, they obviously would also use memory. For this reason, there is the possibility to use a custom \_cache\_key on all the filters (including the terms filter) where you can control the key under which the filter will be used. Can you use that in someway? When we added the \_cache\_key option, one other option was to have an option and use something like md5 on the terms filters values to automatically generate a cache key, but its not in yet... -shay.banon On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad \< [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com)\> wrote: > Hi Shay, > I am allocating 6G to the ES instance and am using quite a lot of > terms filters. A single > call to a filter should although not have too many terms (30 > non-analyzed urls). > > Is there a way to limit the size the cache can take? By reading the > documentation I thought that > the cache would by default max to 20% of the heap size, but in my case > the cache is at 3.8GB > if I understand it correctly. > > Best Regards, > Michel > > On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > How much memory do you allocate to the instance? This memory reference to > > the filter cache, are you maybe using large terms filters with a lot of > > values? > > > > On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > Hi, > > > lately I am getting OOM Exceptions in ES and I suspect that it might > > > have something to do with requests I'm sending to the ES cluster. > > > > > > I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap > > > takes 3.8 GB. > > > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > > > > > Is there a possibility to log the queries the individual nodes handle, > > > so I might see what happens immediately before the OOM? > > > > > > Thanks, > > > Michel ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [September 26, 2011, 9:06am UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/5 "2011-09-26T09:06:47Z")
Hi Shay, I had another look at the heap dump regarding your comments, and for me it looks as if the FilterCacheValues take up over 3GB, therefore I still don't understand why the cache gets so big. I pasted a dominator tree of the filtercache grouped by class. > Best, Michel On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Hi, > It might be related to the filters you use then. The filters cache has > two aspects to it, the cache key and the values. The memory limit on it is > computed against the values. Its quite difficult to compute the memory used > for the keys (which are the filters themselves), but, they obviously would > also use memory. For this reason, there is the possibility to use a custom > \_cache\_key on all the filters (including the terms filter) where you can > control the key under which the filter will be used. Can you use that in > someway? > When we added the \_cache\_key option, one other option was to have an > option and use something like md5 on the terms filters values to > automatically generate a cache key, but its not in yet... > -shay.banon > > On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > Hi Shay, > > I am allocating 6G to the ES instance and am using quite a lot of > > terms filters. A single > > call to a filter should although not have too many terms (30 > > non-analyzed urls). > > > > Is there a way to limit the size the cache can take? By reading the > > documentation I thought that > > the cache would by default max to 20% of the heap size, but in my case > > the cache is at 3.8GB > > if I understand it correctly. > > > > Best Regards, > > Michel > > > > On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > How much memory do you allocate to the instance? This memory reference > > > to > > > the filter cache, are you maybe using large terms filters with a lot of > > > values? > > > > > > On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad > > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > > > Hi, > > > > lately I am getting OOM Exceptions in ES and I suspect that it might > > > > have something to do with requests I'm sending to the ES cluster. > > > > > > > > I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap > > > > takes 3.8 GB. > > > > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > > > > > > > Is there a possibility to log the queries the individual nodes handle, > > > > so I might see what happens immediately before the OOM? > > > > > > > > Thanks, > > > > Michel ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 26, 2011, 9:35am UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/6 "2011-09-26T09:35:03Z")
What are you using to navigate the heap dump? Is it the stats for live objects only? I ran a quick test and I can see the 20% threshold nicely maintained even when profiling (putting aside cache keys, as I explained before). On Mon, Sep 26, 2011 at 12:06 PM, Michel Conrad \< [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com)\> wrote: > Hi Shay, > > I had another look at the heap dump regarding your comments, > and for me it looks as if the FilterCacheValues take up over 3GB, > therefore I still don't understand why the cache gets so big. > > I pasted a dominator tree of the filtercache grouped by class. > [gist:eaaa3df2e5c3f8f62dc1 · GitHub](https://gist.github.com/eaaa3df2e5c3f8f62dc1) > > Best, > Michel > > On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Hi, > > It might be related to the filters you use then. The filters cache has > > two aspects to it, the cache key and the values. The memory limit on it > > is > > computed against the values. Its quite difficult to compute the memory > > used > > for the keys (which are the filters themselves), but, they obviously > > would > > also use memory. For this reason, there is the possibility to use a > > custom > > \_cache\_key on all the filters (including the terms filter) where you can > > control the key under which the filter will be used. Can you use that in > > someway? > > When we added the \_cache\_key option, one other option was to have an > > option and use something like md5 on the terms filters values to > > automatically generate a cache key, but its not in yet... > > -shay.banon > > > > On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > Hi Shay, > > > I am allocating 6G to the ES instance and am using quite a lot of > > > terms filters. A single > > > call to a filter should although not have too many terms (30 > > > non-analyzed urls). > > > > > > Is there a way to limit the size the cache can take? By reading the > > > documentation I thought that > > > the cache would by default max to 20% of the heap size, but in my case > > > the cache is at 3.8GB > > > if I understand it correctly. > > > > > > Best Regards, > > > Michel > > > > > > On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > How much memory do you allocate to the instance? This memory reference > > > > to > > > > the filter cache, are you maybe using large terms filters with a lot > > > > of > > > > values? > > > > > > > > On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad > > > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > > > > > Hi, > > > > > lately I am getting OOM Exceptions in ES and I suspect that it might > > > > > have something to do with requests I'm sending to the ES cluster. > > > > > > > > > > I pasted part of the heapdump I got, a single ConcurrentLinkedHashMap > > > > > takes 3.8 GB. > > > > > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > > > > > > > > > Is there a possibility to log the queries the individual nodes > > > > > handle, > > > > > so I might see what happens immediately before the OOM? > > > > > > > > > > Thanks, > > > > > Michel ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [September 26, 2011, 11:16am UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/7 "2011-09-26T11:16:03Z")
Hi, I'm using the Eclipse Memory Analyzer. I click on the ConcurrentLinkedHashMap / Java Basics / Open in dominator tree with grouping the objects by class. This is where I get over 3Gb retained heap for FilterCacheValues from. Best, Michel On Mon, Sep 26, 2011 at 11:35 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > What are you using to navigate the heap dump? Is it the stats for live > objects only? I ran a quick test and I can see the 20% threshold nicely > maintained even when profiling (putting aside cache keys, as I explained > before). > > On Mon, Sep 26, 2011 at 12:06 PM, Michel Conrad > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > Hi Shay, > > > > I had another look at the heap dump regarding your comments, > > and for me it looks as if the FilterCacheValues take up over 3GB, > > therefore I still don't understand why the cache gets so big. > > > > I pasted a dominator tree of the filtercache grouped by class. > > [gist:eaaa3df2e5c3f8f62dc1 · GitHub](https://gist.github.com/eaaa3df2e5c3f8f62dc1) > > > > Best, > > Michel > > > > On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > Hi, > > > It might be related to the filters you use then. The filters cache > > > has > > > two aspects to it, the cache key and the values. The memory limit on it > > > is > > > computed against the values. Its quite difficult to compute the memory > > > used > > > for the keys (which are the filters themselves), but, they obviously > > > would > > > also use memory. For this reason, there is the possibility to use a > > > custom > > > \_cache\_key on all the filters (including the terms filter) where you can > > > control the key under which the filter will be used. Can you use that in > > > someway? > > > When we added the \_cache\_key option, one other option was to have an > > > option and use something like md5 on the terms filters values to > > > automatically generate a cache key, but its not in yet... > > > -shay.banon > > > > > > On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad > > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > > > Hi Shay, > > > > I am allocating 6G to the ES instance and am using quite a lot of > > > > terms filters. A single > > > > call to a filter should although not have too many terms (30 > > > > non-analyzed urls). > > > > > > > > Is there a way to limit the size the cache can take? By reading the > > > > documentation I thought that > > > > the cache would by default max to 20% of the heap size, but in my case > > > > the cache is at 3.8GB > > > > if I understand it correctly. > > > > > > > > Best Regards, > > > > Michel > > > > > > > > On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > How much memory do you allocate to the instance? This memory > > > > > reference > > > > > to > > > > > the filter cache, are you maybe using large terms filters with a lot > > > > > of > > > > > values? > > > > > > > > > > On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad > > > > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > > > > > > > Hi, > > > > > > lately I am getting OOM Exceptions in ES and I suspect that it might > > > > > > have something to do with requests I'm sending to the ES cluster. > > > > > > > > > > > > I pasted part of the heapdump I got, a single > > > > > > ConcurrentLinkedHashMap > > > > > > takes 3.8 GB. > > > > > > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > > > > > > > > > > > Is there a possibility to log the queries the individual nodes > > > > > > handle, > > > > > > so I might see what happens immediately before the OOM? > > > > > > > > > > > > Thanks, > > > > > > Michel ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 26, 2011, 3:59pm UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/8 "2011-09-26T15:59:51Z")
Haven't used EMA for a long time, so I don't know if what you do excludes uncollected references. On Mon, Sep 26, 2011 at 2:16 PM, Michel Conrad \< [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com)\> wrote: > Hi, > I'm using the Eclipse Memory Analyzer. > I click on the ConcurrentLinkedHashMap / Java Basics / Open in > dominator tree with grouping the objects by class. > This is where I get over 3Gb retained heap for FilterCacheValues from. > > Best, > Michel > > On Mon, Sep 26, 2011 at 11:35 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > What are you using to navigate the heap dump? Is it the stats for live > > objects only? I ran a quick test and I can see the 20% threshold nicely > > maintained even when profiling (putting aside cache keys, as I explained > > before). > > > > On Mon, Sep 26, 2011 at 12:06 PM, Michel Conrad > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > Hi Shay, > > > > > > I had another look at the heap dump regarding your comments, > > > and for me it looks as if the FilterCacheValues take up over 3GB, > > > therefore I still don't understand why the cache gets so big. > > > > > > I pasted a dominator tree of the filtercache grouped by class. > > > [gist:eaaa3df2e5c3f8f62dc1 · GitHub](https://gist.github.com/eaaa3df2e5c3f8f62dc1) > > > > > > Best, > > > Michel > > > > > > On Mon, Sep 26, 2011 at 10:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > Hi, > > > > It might be related to the filters you use then. The filters cache > > > > has > > > > two aspects to it, the cache key and the values. The memory limit on > > > > it > > > > is > > > > computed against the values. Its quite difficult to compute the memory > > > > used > > > > for the keys (which are the filters themselves), but, they obviously > > > > would > > > > also use memory. For this reason, there is the possibility to use a > > > > custom > > > > \_cache\_key on all the filters (including the terms filter) where you > > > > can > > > > control the key under which the filter will be used. Can you use that > > > > in > > > > someway? > > > > When we added the \_cache\_key option, one other option was to have an > > > > option and use something like md5 on the terms filters values to > > > > automatically generate a cache key, but its not in yet... > > > > -shay.banon > > > > > > > > On Mon, Sep 26, 2011 at 10:52 AM, Michel Conrad > > > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > > > > > Hi Shay, > > > > > I am allocating 6G to the ES instance and am using quite a lot of > > > > > terms filters. A single > > > > > call to a filter should although not have too many terms (30 > > > > > non-analyzed urls). > > > > > > > > > > Is there a way to limit the size the cache can take? By reading the > > > > > documentation I thought that > > > > > the cache would by default max to 20% of the heap size, but in my > > > > > case > > > > > the cache is at 3.8GB > > > > > if I understand it correctly. > > > > > > > > > > Best Regards, > > > > > Michel > > > > > > > > > > On Sat, Sep 24, 2011 at 1:12 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > > > > > wrote: > > > > > > > > > > > How much memory do you allocate to the instance? This memory > > > > > > reference > > > > > > to > > > > > > the filter cache, are you maybe using large terms filters with a > > > > > > lot > > > > > > of > > > > > > values? > > > > > > > > > > > > On Fri, Sep 23, 2011 at 4:31 PM, Michel Conrad > > > > > > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > > > > > > > > > > > Hi, > > > > > > > lately I am getting OOM Exceptions in ES and I suspect that it > > > > > > > might > > > > > > > have something to do with requests I'm sending to the ES cluster. > > > > > > > > > > > > > > I pasted part of the heapdump I got, a single > > > > > > > ConcurrentLinkedHashMap > > > > > > > takes 3.8 GB. > > > > > > > [gist:1237322 · GitHub](https://gist.github.com/1237322) > > > > > > > > > > > > > > Is there a possibility to log the queries the individual nodes > > > > > > > handle, > > > > > > > so I might see what happens immediately before the OOM? > > > > > > > > > > > > > > Thanks, > > > > > > > Michel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:53am UTC](https://discuss.elastic.co/t/elasticsearch-outofmemory-exceptions/5441/9 "2017-07-06T03:53:35Z")