# JVM Exception when searching on huge data

**URL:** <https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579>\
**Category:** Elasticsearch\
**Created:** [May 7, 2012, 12:58pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579 "2012-05-07T12:58:28Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\
**Post date:** [May 7, 2012, 12:58pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/1 "2012-05-07T12:58:28Z")

</div>

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array  
Types. As an addition of data, we introduced Nested and Array Types in to  
the existing index. This new fields carry even an array size of 100. When  
I try to query now after indexing data, its bringing the below JVM memory.  
Already we have an JVM of

set.default.ES\_MIN\_MEM=2048  
set.default.ES\_MAX\_MEM=8192

My App was working fine in local when with smaller data(consists of nested  
and array index too). Today I started to run the app with a huge data. And  
when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak]  
[2012-05-03] loading field [field.field\_type\_and\_value] caused out of  
memory failure  
java.lang.OutOfMemoryError: Java heap space  
at  
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)  
at  
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)  
at  
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)  
at  
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)  
at  
org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)  
at  
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)  
at  
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)  
at  
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)  
at  
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)  
at  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)  
at  
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)  
at  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)  
at  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)  
at  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)  
at  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)  
at  
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)  
at  
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)  
at  
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
[2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
[717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
Received response for a request that has timed out, sent [31974ms] ago,  
timed out [1973ms] ago, action [discovery/zen/fd/ping], node  
[[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\
**Post date:** [May 7, 2012, 1:02pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/2 "2012-05-07T13:02:30Z")

</div>

It gone more weird....

Now the Index status turned Yellow too...

"cluster\_name" : "test",  
"status" : "yellow",  
"timed\_out" : false,  
"number\_of\_nodes" : 2,  
"number\_of\_data\_nodes" : 2,  
"active\_primary\_shards" : 277,  
"active\_shards" : 461,  
"relocating\_shards" : 0,  
"initializing\_shards" : 2,  
"unassigned\_shards" : 91

On Monday, May 7, 2012 6:28:28 PM UTC+5:30, Manoj wrote:

> Hi Shay,
> 
> We were running an App on Elasticsearch, which didn't have Nested or Array  
> Types. As an addition of data, we introduced Nested and Array Types in to  
> the existing index. This new fields carry even an array size of 100. When  
> I try to query now after indexing data, its bringing the below JVM memory.  
> Already we have an JVM of
> 
> set.default.ES\_MIN\_MEM=2048  
> set.default.ES\_MAX\_MEM=8192
> 
> My App was working fine in local when with smaller data(consists of nested  
> and array index too). Today I started to run the app with a huge data. And  
> when on searching this huge index the following exception is occurring...
> 
> [2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak]  
> [2012-05-03] loading field [field.field\_type\_and\_value] caused out of  
> memory failure  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)  
> at  
> org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)  
> at  
> org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)  
> at  
> org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)  
> at  
> org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)  
> at  
> org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)  
> at  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)  
> at  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)  
> at  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> 
> [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> Received response for a request that has timed out, sent [31974ms] ago,  
> timed out [1973ms] ago, action [discovery/zen/fd/ping], node  
> [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\
**Post date:** [May 7, 2012, 1:07pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/3 "2012-05-07T13:07:31Z")

</div>

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES\_MIN\_MEM=2048

set.default.ES\_MAX\_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

```
    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)

    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)

    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)

    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)

    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)

    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)

    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)

    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)

    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

```

[2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN][transport] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\
**Post date:** [May 7, 2012, 1:15pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/4 "2012-05-07T13:15:00Z")

</div>

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

> Hello!
> 
> That error tells you that you don't have enough memory to load field data  
> into field data cache. Do you use faceting or sorting ?
> 
> \*--  
> Regards,  
> Rafał Kuć  
> Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> 
> - 
> 
> Hi Shay,
> 
> We were running an App on Elasticsearch, which didn't have Nested or Array  
> Types. As an addition of data, we introduced Nested and Array Types in to  
> the existing index. This new fields carry even an array size of 100. When  
> I try to query now after indexing data, its bringing the below JVM memory.  
> Already we have an JVM of
> 
> set.default.ES\_MIN\_MEM=2048  
> set.default.ES\_MAX\_MEM=8192
> 
> My App was working fine in local when with smaller data(consists of nested  
> and array index too). Today I started to run the app with a huge data. And  
> when on searching this huge index the following exception is occurring...
> 
> [2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak]  
> [2012-05-03] loading field [field.field\_type\_and\_value] caused out of  
> memory failure  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)  
> at  
> org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)  
> at  
> org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)  
> at  
> org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)  
> at  
> org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)  
> at  
> org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)  
> at  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)  
> at  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)  
> at  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> 
> [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> Received response for a request that has timed out, sent [31974ms] ago,  
> timed out [1973ms] ago, action [discovery/zen/fd/ping], node  
> [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\
**Post date:** [May 7, 2012, 1:24pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/5 "2012-05-07T13:24:31Z")

</div>

Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: [https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk) , because it's more or less about the same situation you are currently experiencing.

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES\_MIN\_MEM=2048

set.default.ES\_MAX\_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

```
    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)

    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)

    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)

    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)

    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)

    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)

    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)

    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)

    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

```

[2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN][transport] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\
**Post date:** [May 7, 2012, 2:27pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/6 "2012-05-07T14:27:46Z")

</div>

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for  
index.cache.field.max\_size: 10000

-FYI-  
while running the Nodes stats we found the following detail  
"size" : "794.7gb",

"cache" : {  
"field\_evictions" : 0,  
"field\_size" : "6.4gb",  
"field\_size\_in\_bytes" : 6899525304,  
"filter\_count" : 325,  
"filter\_evictions" : 0,  
"filter\_size" : "55.7mb",  
"filter\_size\_in\_bytes" : 58422192  
},

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:

> Hello!
> 
> There was a thread recently about field data cache and its memory usage.  
> Basically, when you use faceting or sorting on a given field for the first  
> time, Elasticsearch needs to read data for that field and put it into field  
> data cache. That's why you didn't encounter that problem with small amount  
> of data and you started seeing it after the amount of data changed.
> 
> What you can do is limit your field data cache size, set it expiration  
> time or change its type from resident to soft. I suggest to read the  
> following thread:  
> [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk) ,  
> because it's more or less about the same situation you are currently  
> experiencing.
> 
> \*--  
> Regards,  
> Rafał Kuć  
> Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> 
> - 
> 
> Thanks for the fast reply Rafal .....
> 
> We are using Faceting on that field
> 
> Could this JVM cause the index to go Yellow?
> 
> On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:  
> Hello!
> 
> That error tells you that you don't have enough memory to load field data  
> into field data cache. Do you use faceting or sorting ?
> 
> \*--  
> Regards,  
> Rafał Kuć  
> Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> 
> - 
> 
> Hi Shay,
> 
> We were running an App on Elasticsearch, which didn't have Nested or Array  
> Types. As an addition of data, we introduced Nested and Array Types in to  
> the existing index. This new fields carry even an array size of 100. When  
> I try to query now after indexing data, its bringing the below JVM memory.  
> Already we have an JVM of
> 
> set.default.ES\_MIN\_MEM=2048  
> set.default.ES\_MAX\_MEM=8192
> 
> My App was working fine in local when with smaller data(consists of nested  
> and array index too). Today I started to run the app with a huge data. And  
> when on searching this huge index the following exception is occurring...
> 
> [2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak]  
> [2012-05-03] loading field [field.field\_type\_and\_value] caused out of  
> memory failure  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)  
> at  
> org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)  
> at  
> org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)  
> at  
> org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)  
> at  
> org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)  
> at  
> org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)  
> at  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)  
> at  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)  
> at  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> 
> [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> Received response for a request that has timed out, sent [31974ms] ago,  
> timed out [1973ms] ago, action [discovery/zen/fd/ping], node  
> [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\
**Post date:** [May 7, 2012, 2:51pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/7 "2012-05-07T14:51:34Z")

</div>

Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error.

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES\_MAX\_MEM set to 8gb you are probably near the end of the heap limit.

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for

index.cache.field.max\_size: 10000

-FYI-

while running the Nodes stats we found the following detail

"size" : "794.7gb",

"cache" : {

```
      "field_evictions" : 0,

      "field_size" : "6.4gb",

      "field_size_in_bytes" : 6899525304,

      "filter_count" : 325,

      "filter_evictions" : 0,

      "filter_size" : "55.7mb",

      "filter_size_in_bytes" : 58422192

    },

```

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: [https://groups.google.](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)[com/forum/?fromgroups#!topic/](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)[elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk) , because it's more or less about the same situation you are currently experiencing.

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES\_MIN\_MEM=2048

set.default.ES\_MAX\_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

```
    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)

    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)

    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)

    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)

    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)

    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)

    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)

    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)

    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

```

[2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN][transport] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\
**Post date:** [May 7, 2012, 3:45pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/8 "2012-05-07T15:45:34Z")

</div>

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

1. /usr/share/es/bin/service/elasticsearch.conf
2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max\_size and  
index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

> Hello!
> 
> Yes, this may be the reason your cluster went from Green to Yellow state,  
> for example one of your nodes got disconnected from the cluster because of  
> the garbage collection which happened before JVM reported OutOfMemory  
> error.
> 
> I can't say how many entries in the field data cache will be OK for you.  
> You may need to experiment with that. In the stats you pasted, you can see  
> that while doing the stats, your cache was already at 6.4gb for that node.  
> Given that you have ES\_MAX\_MEM set to 8gb you are probably near the end of  
> the heap limit.
> 
> \*--  
> Regards,  
> Rafał Kuć  
> Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> 
> - 
> 
> Thanks Rafal, you showed me the right way...
> 
> Could u also help me in finding answers for the below:
> 
> Is this the reason for the index to go Yellow?
> 
> How can we decide up on the size to be allotted for  
> index.cache.field.max\_size: 10000
> 
> -FYI-  
> while running the Nodes stats we found the following detail  
> "size" : "794.7gb",
> 
> "cache" : {  
> "field\_evictions" : 0,  
> "field\_size" : "6.4gb",  
> "field\_size\_in\_bytes" : 6899525304,  
> "filter\_count" : 325,  
> "filter\_evictions" : 0,  
> "filter\_size" : "55.7mb",  
> "filter\_size\_in\_bytes" : 58422192  
> },
> 
> Thank You!
> 
> On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:  
> Hello!
> 
> There was a thread recently about field data cache and its memory usage.  
> Basically, when you use faceting or sorting on a given field for the first  
> time, Elasticsearch needs to read data for that field and put it into field  
> data cache. That's why you didn't encounter that problem with small amount  
> of data and you started seeing it after the amount of data changed.
> 
> What you can do is limit your field data cache size, set it expiration  
> time or change its type from resident to soft. I suggest to read the  
> following thread: [https://groups.google](https://groups.google).[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> com/forum/?fromgroups#!topic/[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> elasticsearch/DLcErfeivQk[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk) ,  
> because it's more or less about the same situation you are currently  
> experiencing.
> 
> \*--  
> Regards,  
> Rafał Kuć  
> Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> 
> - 
> 
> Thanks for the fast reply Rafal .....
> 
> We are using Faceting on that field
> 
> Could this JVM cause the index to go Yellow?
> 
> On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:  
> Hello!
> 
> That error tells you that you don't have enough memory to load field data  
> into field data cache. Do you use faceting or sorting ?
> 
> \*--  
> Regards,  
> Rafał Kuć  
> Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> 
> - 
> 
> Hi Shay,
> 
> We were running an App on Elasticsearch, which didn't have Nested or Array  
> Types. As an addition of data, we introduced Nested and Array Types in to  
> the existing index. This new fields carry even an array size of 100. When  
> I try to query now after indexing data, its bringing the below JVM memory.  
> Already we have an JVM of
> 
> set.default.ES\_MIN\_MEM=2048  
> set.default.ES\_MAX\_MEM=8192
> 
> My App was working fine in local when with smaller data(consists of nested  
> and array index too). Today I started to run the app with a huge data. And  
> when on searching this huge index the following exception is occurring...
> 
> [2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak]  
> [2012-05-03] loading field [field.field\_type\_and\_value] caused out of  
> memory failure  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)  
> at  
> org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)  
> at  
> org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)  
> at  
> org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)  
> at  
> org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)  
> at  
> org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)  
> at  
> org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)  
> at  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)  
> at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)  
> at  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)  
> at  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> 
> [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> Received response for a request that has timed out, sent [31974ms] ago,  
> timed out [1973ms] ago, action [discovery/zen/fd/ping], node  
> [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\
**Post date:** [May 7, 2012, 3:50pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/9 "2012-05-07T15:50:13Z")

</div>

Hello!

You need to check which of those two files is the one that is actually used in your system (maybe one is a link to the other ?).

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

1. /usr/share/es/bin/service/elasticsearch.conf

2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max\_size and index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error.

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES\_MAX\_MEM set to 8gb you are probably near the end of the heap limit.

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for

index.cache.field.max\_size: 10000

-FYI-

while running the Nodes stats we found the following detail

"size" : "794.7gb",

"cache" : {

```
      "field_evictions" : 0,

      "field_size" : "6.4gb",

      "field_size_in_bytes" : 6899525304,

      "filter_count" : 325,

      "filter_evictions" : 0,

      "filter_size" : "55.7mb",

      "filter_size_in_bytes" : 58422192

    },

```

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: [https://groups.google.](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)[com/forum/?fromgroups#!topic/](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)[e](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)[lasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk) , because it's more or less about the same situation you are currently experiencing.

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch_

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES\_MIN\_MEM=2048

set.default.ES\_MAX\_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

```
    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)

    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)

    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)

    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)

    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)

    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)

    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)

    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)

    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)

    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

```

[2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN][transport] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [May 9, 2012, 9:04am UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/10 "2012-05-09T09:04:09Z")

</div>

Its better to simply increase the memory used for the JVM, or scale out to  
more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj [manokrrish@gmail.com](mailto:manokrrish@gmail.com) wrote:

> Thanks Shay, that really helped...
> 
> I could find two configuration file in linux installed Elasticsearch
> 
> 1. /usr/share/es/bin/service/elasticsearch.conf
> 2. another is es/config/elasticsearch.yml
> 
> Whr should I put this configuration index.cache.field.max\_size and  
> index.cache.field.type?
> 
> On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
> 
> > Hello!
> > 
> > Yes, this may be the reason your cluster went from Green to Yellow state,  
> > for example one of your nodes got disconnected from the cluster because of  
> > the garbage collection which happened before JVM reported OutOfMemory  
> > error.
> > 
> > I can't say how many entries in the field data cache will be OK for you.  
> > You may need to experiment with that. In the stats you pasted, you can see  
> > that while doing the stats, your cache was already at 6.4gb for that node.  
> > Given that you have ES\_MAX\_MEM set to 8gb you are probably near the end of  
> > the heap limit.
> > 
> > \*--  
> > Regards,  
> > Rafał Kuć  
> > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > 
> > - 
> > 
> > Thanks Rafal, you showed me the right way...
> > 
> > Could u also help me in finding answers for the below:
> > 
> > Is this the reason for the index to go Yellow?
> > 
> > How can we decide up on the size to be allotted for  
> > index.cache.field.max\_size: 10000
> > 
> > -FYI-  
> > while running the Nodes stats we found the following detail  
> > "size" : "794.7gb",
> > 
> > "cache" : {  
> > "field\_evictions" : 0,  
> > "field\_size" : "6.4gb",  
> > "field\_size\_in\_bytes" : 6899525304,  
> > "filter\_count" : 325,  
> > "filter\_evictions" : 0,  
> > "filter\_size" : "55.7mb",  
> > "filter\_size\_in\_bytes" : 58422192  
> > },
> > 
> > Thank You!
> > 
> > On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:  
> > Hello!
> > 
> > There was a thread recently about field data cache and its memory usage.  
> > Basically, when you use faceting or sorting on a given field for the first  
> > time, Elasticsearch needs to read data for that field and put it into field  
> > data cache. That's why you didn't encounter that problem with small amount  
> > of data and you started seeing it after the amount of data changed.
> > 
> > What you can do is limit your field data cache size, set it expiration  
> > time or change its type from resident to soft. I suggest to read the  
> > following thread: [https://groups.google](https://groups.google).[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> > **com/forum/?fromgroups#!topic/[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> > e**lasticsearch/DLcErfeivQk[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk) ,  
> > because it's more or less about the same situation you are currently  
> > experiencing.
> > 
> > \*--  
> > Regards,  
> > Rafał Kuć  
> > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > 
> > - 
> > 
> > Thanks for the fast reply Rafal .....
> > 
> > We are using Faceting on that field
> > 
> > Could this JVM cause the index to go Yellow?
> > 
> > On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:  
> > Hello!
> > 
> > That error tells you that you don't have enough memory to load field data  
> > into field data cache. Do you use faceting or sorting ?
> > 
> > \*--  
> > Regards,  
> > Rafał Kuć  
> > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > 
> > - 
> > 
> > Hi Shay,
> > 
> > We were running an App on Elasticsearch, which didn't have Nested or  
> > Array Types. As an addition of data, we introduced Nested and Array Types  
> > in to the existing index. This new fields carry even an array size of 100.  
> > When I try to query now after indexing data, its bringing the below JVM  
> > memory. Already we have an JVM of
> > 
> > set.default.ES\_MIN\_MEM=2048  
> > set.default.ES\_MAX\_MEM=8192
> > 
> > My App was working fine in local when with smaller data(consists of  
> > nested and array index too). Today I started to run the app with a huge  
> > data. And when on searching this huge index the following exception is  
> > occurring...
> > 
> > [2012-05-07 00:47:50,403][WARN][index.cache.field.data.\*_resident]  
> > [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out  
> > of memory failure  
> > java.lang.OutOfMemoryError: Java heap space  
> > at org.elasticsearch.index.field.data.support.FieldDataLoader.  
> > \*load(FieldDataLoader.java:61)  
> > at org.elasticsearch.index.field.data.strings.StringFieldData.  
> > load(StringFieldData.java:90)  
> > at org.elasticsearch.index.field. **data.strings.**  
> > StringFieldDataType.load(**StringFieldDataType.java:56)  
> > at org.elasticsearch.index.field.data.strings.  
> > StringFieldDataType.load(StringFieldDataType.java:34)  
> > at org.elasticsearch.index.field.data.FieldData.load(FieldData.  
> > java:111)  
> > at org.elasticsearch.index.cache.field.data.support.  
> > AbstractConcurrentMapFieldDataCache.cache(  
> > AbstractConcurrentMapFieldDataCache.java:122)  
> > at org.elasticsearch.search.facet.terms.strings.  
> > TermsStringOrdinalsFacetCollec**tor.doSetNextReader(  
> > TermsStringOrdinalsFacetCollec_tor.java:128)  
> > at org.elasticsearch.search. **facet.AbstractFacetCollector.**  
> > setNextReader(\*\*AbstractFacetCollector.java:\*\*75)  
> > at org.elasticsearch.common. **lucene.MultiCollector.**  
> > setNextReader(MultiCollector.\*\*java:67)  
> > at org.apache.lucene.search.**IndexSearcher.search(**  
> > IndexSearcher.java:576)  
> > at org.elasticsearch.search. **internal.ContextIndexSearcher.**  
> > search(ContextIndexSearcher.\*\*java:195)  
> > at org.apache.lucene.search.**IndexSearcher.search(**  
> > IndexSearcher.java:445)  
> > at org.apache.lucene.search.**IndexSearcher.search(**  
> > IndexSearcher.java:426)  
> > at org.apache.lucene.search.**IndexSearcher.search(**  
> > IndexSearcher.java:342)  
> > at org.apache.lucene.search.**IndexSearcher.search(**  
> > IndexSearcher.java:330)  
> > at org.elasticsearch.search.**query.QueryPhase.execute(**  
> > QueryPhase.java:194)  
> > at org.elasticsearch.search.\*\*SearchService._executeQueryPhase(  
> > SearchService.java:234)  
> > at org.elasticsearch.search. **action.**  
> > SearchServiceTransportAction.**sendExecuteQuery(**  
> > SearchServiceTransportAction.**java:140)  
> > at org.elasticsearch.action.search.type.  
> > TransportSearchQueryThenFetchA**ction$AsyncAction._  
> > sendExecuteFirstPhase( **TransportSearchQueryThenFetchA** ction.java:80)  
> > at org.elasticsearch.action. **search.type.**  
> > TransportSearchTypeAction$\*\*BaseAsyncAction.**performFirstPhase(**  
> > TransportSearchTypeAction.\*\*java:204)  
> > at org.elasticsearch.action. **search.type.**  
> > TransportSearchTypeAction$\*\*BaseAsyncAction.**performFirstPhase(**  
> > TransportSearchTypeAction.\*\*java:191)  
> > at org.elasticsearch.action. **search.type.**  
> > TransportSearchTypeAction$**BaseAsyncAction$2.run(**  
> > TransportSearchTypeAction.\*\*java:177)  
> > at java.util.concurrent. **ThreadPoolExecutor$Worker.**  
> > runTask(ThreadPoolExecutor.\*\*java:886)  
> > at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**  
> > ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.\*\*java:662)  
> > [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> > [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> > [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> > 
> > [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> > Received response for a request that has timed out, sent [31974ms] ago,  
> > timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][\*\*  
> > tlPRmwDxQ0ma7PJJPD2oXw][inet[/\*\*172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\
**Post date:** [May 15, 2012, 4:32am UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/11 "2012-05-15T04:32:54Z")

</div>

Thanks Kimchy and Rafal....

We increased the JVM memory to 16GB. Also changed the Array type fields in  
the document to String types. Achieved String type by using some  
separating special characters between. Now, with this new String type data  
of about 50GB, we again went for faceting. It still causing the problem...

Also, I would like to confirm, if we are using enough the machine's  
efficiency by using 2 machines now, with 16GB of JVM for ES. What is the  
MAX JVM I can give for the current structure?

I am just interested to know the MAX efficiency that could be pulled out of  
a cluster....

Thanks a lot!

On Wednesday, May 9, 2012 2:34:09 PM UTC+5:30, kimchy wrote:

> Its better to simply increase the memory used for the JVM, or scale out to  
> more machines in the cluster.
> 
> On Mon, May 7, 2012 at 6:45 PM, Manoj [manokrrish@gmail.com](mailto:manokrrish@gmail.com) wrote:
> 
> > Thanks Shay, that really helped...
> > 
> > I could find two configuration file in linux installed Elasticsearch
> > 
> > 1. /usr/share/es/bin/service/elasticsearch.conf
> > 2. another is es/config/elasticsearch.yml
> > 
> > Whr should I put this configuration index.cache.field.max\_size and  
> > index.cache.field.type?
> > 
> > On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
> > 
> > > Hello!
> > > 
> > > Yes, this may be the reason your cluster went from Green to Yellow  
> > > state, for example one of your nodes got disconnected from the cluster  
> > > because of the garbage collection which happened before JVM reported  
> > > OutOfMemory error.
> > > 
> > > I can't say how many entries in the field data cache will be OK for you.  
> > > You may need to experiment with that. In the stats you pasted, you can see  
> > > that while doing the stats, your cache was already at 6.4gb for that node.  
> > > Given that you have ES\_MAX\_MEM set to 8gb you are probably near the end of  
> > > the heap limit.
> > > 
> > > \*--  
> > > Regards,  
> > > Rafał Kuć  
> > > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > > 
> > > - 
> > > 
> > > Thanks Rafal, you showed me the right way...
> > > 
> > > Could u also help me in finding answers for the below:
> > > 
> > > Is this the reason for the index to go Yellow?
> > > 
> > > How can we decide up on the size to be allotted for  
> > > index.cache.field.max\_size: 10000
> > > 
> > > -FYI-  
> > > while running the Nodes stats we found the following detail  
> > > "size" : "794.7gb",
> > > 
> > > "cache" : {  
> > > "field\_evictions" : 0,  
> > > "field\_size" : "6.4gb",  
> > > "field\_size\_in\_bytes" : 6899525304,  
> > > "filter\_count" : 325,  
> > > "filter\_evictions" : 0,  
> > > "filter\_size" : "55.7mb",  
> > > "filter\_size\_in\_bytes" : 58422192  
> > > },
> > > 
> > > Thank You!
> > > 
> > > On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:  
> > > Hello!
> > > 
> > > There was a thread recently about field data cache and its memory usage.  
> > > Basically, when you use faceting or sorting on a given field for the first  
> > > time, Elasticsearch needs to read data for that field and put it into field  
> > > data cache. That's why you didn't encounter that problem with small amount  
> > > of data and you started seeing it after the amount of data changed.
> > > 
> > > What you can do is limit your field data cache size, set it expiration  
> > > time or change its type from resident to soft. I suggest to read the  
> > > following thread: [https://groups.google](https://groups.google).[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> > > **com/forum/?fromgroups#!topic/[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> > > e**lasticsearch/DLcErfeivQk[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk) ,  
> > > because it's more or less about the same situation you are currently  
> > > experiencing.
> > > 
> > > \*--  
> > > Regards,  
> > > Rafał Kuć  
> > > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > > 
> > > - 
> > > 
> > > Thanks for the fast reply Rafal .....
> > > 
> > > We are using Faceting on that field
> > > 
> > > Could this JVM cause the index to go Yellow?
> > > 
> > > On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:  
> > > Hello!
> > > 
> > > That error tells you that you don't have enough memory to load field  
> > > data into field data cache. Do you use faceting or sorting ?
> > > 
> > > \*--  
> > > Regards,  
> > > Rafał Kuć  
> > > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > > 
> > > - 
> > > 
> > > Hi Shay,
> > > 
> > > We were running an App on Elasticsearch, which didn't have Nested or  
> > > Array Types. As an addition of data, we introduced Nested and Array Types  
> > > in to the existing index. This new fields carry even an array size of 100.  
> > > When I try to query now after indexing data, its bringing the below JVM  
> > > memory. Already we have an JVM of
> > > 
> > > set.default.ES\_MIN\_MEM=2048  
> > > set.default.ES\_MAX\_MEM=8192
> > > 
> > > My App was working fine in local when with smaller data(consists of  
> > > nested and array index too). Today I started to run the app with a huge  
> > > data. And when on searching this huge index the following exception is  
> > > occurring...
> > > 
> > > [2012-05-07 00:47:50,403][WARN][index.cache.field.data.\*\*resident]  
> > > [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out  
> > > of memory failure  
> > > java.lang.OutOfMemoryError: Java heap space  
> > > at org.elasticsearch.index.field.**data.support.FieldDataLoader.  
> > > load(FieldDataLoader.java:61)  
> > > at org.elasticsearch.index.field.data.strings.StringFieldData.  
> > > load(StringFieldData.java:90)  
> > > at org.elasticsearch.index.field.data.strings.  
> > > StringFieldDataType.load(StringFieldDataType.java:56)  
> > > at org.elasticsearch.index.field.data.strings.  
> > > StringFieldDataType.load(StringFieldDataType.java:34)  
> > > at org.elasticsearch.index.field.  
> > > data.FieldData.load(FieldData.java:111)  
> > > at org.elasticsearch.index.cache.field.data.support.  
> > > AbstractConcurrentMapFieldDataCache.cache(  
> > > AbstractConcurrentMapFieldDataCache.java:122)  
> > > at org.elasticsearch.search.facet.terms.strings.  
> > > TermsStringOrdinalsFacetCollector.doSetNextReader(  
> > > TermsStringOrdinalsFacetCollec**tor.java:128)  
> > > at org.elasticsearch.search. **facet.AbstractFacetCollector.**  
> > > setNextReader(\*\*AbstractFacetCollector.java:\*\*75)  
> > > at org.elasticsearch.common. **lucene.MultiCollector.**  
> > > setNextReader(MultiCollector.\*\*java:67)  
> > > at org.apache.lucene.search.**IndexSearcher.search(**  
> > > IndexSearcher.java:576)  
> > > at org.elasticsearch.search. **internal.ContextIndexSearcher.**  
> > > search(ContextIndexSearcher.\*\*java:195)  
> > > at org.apache.lucene.search.**IndexSearcher.search(**  
> > > IndexSearcher.java:445)  
> > > at org.apache.lucene.search.**IndexSearcher.search(**  
> > > IndexSearcher.java:426)  
> > > at org.apache.lucene.search.**IndexSearcher.search(**  
> > > IndexSearcher.java:342)  
> > > at org.apache.lucene.search.**IndexSearcher.search(**  
> > > IndexSearcher.java:330)  
> > > at org.elasticsearch.search.**query.QueryPhase.execute(**  
> > > QueryPhase.java:194)  
> > > at org.elasticsearch.search.\*\*SearchService.\*\*executeQueryPhase(  
> > > **SearchService.java:234)  
> > > at org.elasticsearch.search.action.  
> > > SearchServiceTransportAction.sendExecuteQuery(  
> > > SearchServiceTransportAction.java:140)  
> > > at org.elasticsearch.action.search.type.  
> > > TransportSearchQueryThenFetchAction$AsyncAction.**  
> > > sendExecuteFirstPhase( **TransportSearchQueryThenFetchA** ction.java:80)  
> > > at org.elasticsearch.action. **search.type.**  
> > > TransportSearchTypeAction$\*\*BaseAsyncAction.**performFirstPhase(**  
> > > TransportSearchTypeAction.\*\*java:204)  
> > > at org.elasticsearch.action. **search.type.**  
> > > TransportSearchTypeAction$\*\*BaseAsyncAction.**performFirstPhase(**  
> > > TransportSearchTypeAction.\*\*java:191)  
> > > at org.elasticsearch.action. **search.type.**  
> > > TransportSearchTypeAction$**BaseAsyncAction$2.run(**  
> > > TransportSearchTypeAction.\*\*java:177)  
> > > at java.util.concurrent. **ThreadPoolExecutor$Worker.**  
> > > runTask(ThreadPoolExecutor.\*\*java:886)  
> > > at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**  
> > > ThreadPoolExecutor.java:908)  
> > > at java.lang.Thread.run(Thread.\*\*java:662)  
> > > [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> > > [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> > > [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> > > 
> > > [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> > > Received response for a request that has timed out, sent [31974ms] ago,  
> > > timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][\*  
> > > \*tlPRmwDxQ0ma7PJJPD2oXw][inet[/\*\*172.16.144.203:9300]]], id [152791]

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [May 17, 2012, 8:27pm UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/12 "2012-05-17T20:27:42Z")

</div>

Are you asking regarding the maximum amount of memory you can allocate to  
the elasticsearch java process? It shouldn't be more than the machine  
memory, leaving space to the OS, but effectively, you probably want to  
leave some memory as well for things like file system cache.

On Tue, May 15, 2012 at 7:32 AM, Manoj [manokrrish@gmail.com](mailto:manokrrish@gmail.com) wrote:

> Thanks Kimchy and Rafal....
> 
> We increased the JVM memory to 16GB. Also changed the Array type fields in  
> the document to String types. Achieved String type by using some  
> separating special characters between. Now, with this new String type data  
> of about 50GB, we again went for faceting. It still causing the problem...
> 
> Also, I would like to confirm, if we are using enough the machine's  
> efficiency by using 2 machines now, with 16GB of JVM for ES. What is the  
> MAX JVM I can give for the current structure?
> 
> I am just interested to know the MAX efficiency that could be pulled out  
> of a cluster....
> 
> Thanks a lot!
> 
> On Wednesday, May 9, 2012 2:34:09 PM UTC+5:30, kimchy wrote:
> 
> > Its better to simply increase the memory used for the JVM, or scale out  
> > to more machines in the cluster.
> > 
> > On Mon, May 7, 2012 at 6:45 PM, Manoj [manokrrish@gmail.com](mailto:manokrrish@gmail.com) wrote:
> > 
> > > Thanks Shay, that really helped...
> > > 
> > > I could find two configuration file in linux installed Elasticsearch
> > > 
> > > 1. /usr/share/es/bin/service/\*\*elasticsearch.conf
> > > 2. another is es/config/elasticsearch.yml
> > > 
> > > Whr should I put this configuration index.cache.field.max\_size and  
> > > index.cache.field.type?
> > > 
> > > On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
> > > 
> > > > Hello!
> > > > 
> > > > Yes, this may be the reason your cluster went from Green to Yellow  
> > > > state, for example one of your nodes got disconnected from the cluster  
> > > > because of the garbage collection which happened before JVM reported  
> > > > OutOfMemory error.
> > > > 
> > > > I can't say how many entries in the field data cache will be OK for  
> > > > you. You may need to experiment with that. In the stats you pasted, you can  
> > > > see that while doing the stats, your cache was already at 6.4gb for that  
> > > > node. Given that you have ES\_MAX\_MEM set to 8gb you are probably near the  
> > > > end of the heap limit.
> > > > 
> > > > \*--  
> > > > Regards,  
> > > > Rafał Kuć  
> > > > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > > > 
> > > > - 
> > > > 
> > > > Thanks Rafal, you showed me the right way...
> > > > 
> > > > Could u also help me in finding answers for the below:
> > > > 
> > > > Is this the reason for the index to go Yellow?
> > > > 
> > > > How can we decide up on the size to be allotted for  
> > > > index.cache.field.max\_size: 10000
> > > > 
> > > > -FYI-  
> > > > while running the Nodes stats we found the following detail  
> > > > "size" : "794.7gb",
> > > > 
> > > > "cache" : {  
> > > > "field\_evictions" : 0,  
> > > > "field\_size" : "6.4gb",  
> > > > "field\_size\_in\_bytes" : 6899525304,  
> > > > "filter\_count" : 325,  
> > > > "filter\_evictions" : 0,  
> > > > "filter\_size" : "55.7mb",  
> > > > "filter\_size\_in\_bytes" : 58422192  
> > > > },
> > > > 
> > > > Thank You!
> > > > 
> > > > On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:  
> > > > Hello!
> > > > 
> > > > There was a thread recently about field data cache and its memory  
> > > > usage. Basically, when you use faceting or sorting on a given field for the  
> > > > first time, Elasticsearch needs to read data for that field and put it into  
> > > > field data cache. That's why you didn't encounter that problem with small  
> > > > amount of data and you started seeing it after the amount of data changed.
> > > > 
> > > > What you can do is limit your field data cache size, set it expiration  
> > > > time or change its type from resident to soft. I suggest to read the  
> > > > following thread: [https://groups.google](https://groups.google).[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> > > > **com/forum/?fromgroups#!topic/[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk)  
> > > > e**lasticsearch/DLcErfeivQk[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk](https://groups.google.com/forum/?fromgroups#%21topic/elasticsearch/DLcErfeivQk) ,  
> > > > because it's more or less about the same situation you are currently  
> > > > experiencing.
> > > > 
> > > > \*--  
> > > > Regards,  
> > > > Rafał Kuć  
> > > > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > > > 
> > > > - 
> > > > 
> > > > Thanks for the fast reply Rafal .....
> > > > 
> > > > We are using Faceting on that field
> > > > 
> > > > Could this JVM cause the index to go Yellow?
> > > > 
> > > > On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:  
> > > > Hello!
> > > > 
> > > > That error tells you that you don't have enough memory to load field  
> > > > data into field data cache. Do you use faceting or sorting ?
> > > > 
> > > > \*--  
> > > > Regards,  
> > > > Rafał Kuć  
> > > > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch
> > > > 
> > > > - 
> > > > 
> > > > Hi Shay,
> > > > 
> > > > We were running an App on Elasticsearch, which didn't have Nested or  
> > > > Array Types. As an addition of data, we introduced Nested and Array Types  
> > > > in to the existing index. This new fields carry even an array size of 100.  
> > > > When I try to query now after indexing data, its bringing the below JVM  
> > > > memory. Already we have an JVM of
> > > > 
> > > > set.default.ES\_MIN\_MEM=2048  
> > > > set.default.ES\_MAX\_MEM=8192
> > > > 
> > > > My App was working fine in local when with smaller data(consists of  
> > > > nested and array index too). Today I started to run the app with a huge  
> > > > data. And when on searching this huge index the following exception is  
> > > > occurring...
> > > > 
> > > > [2012-05-07 00:47:50,403][WARN][index.cache.field.data. **resid** ent]  
> > > > [Cloak] [2012-05-03] loading field [field.field\_type\_and\_value] caused out  
> > > > of memory failure  
> > > > java.lang.OutOfMemoryError: Java heap space  
> > > > at org.elasticsearch.index.field.\*\*\*\*  
> > > > data.support.FieldDataLoader. **l** oad(FieldDataLoader.java:61)  
> > > > at org.elasticsearch.index.field.\*\*\*\*  
> > > > data.strings.StringFieldData. **l** oad(StringFieldData.java:90)  
> > > > at org.elasticsearch.index.field.**data.strings.  
> > > > StringFieldDataType.load(StringFieldDataType.java:56)  
> > > > at org.elasticsearch.index.field.\*\*data.strings.  
> > > > StringFieldDataType.load(StringFieldDataType.java:34)  
> > > > at org.elasticsearch.index.field.  
> > > > data.FieldData.load(FieldData.\*\*\*\*java:111)  
> > > > at org.elasticsearch.index.cache.\*\*field.data.support.  
> > > > AbstractConcurrentMapFieldDataCache.cache(  
> > > > AbstractConcurrentMapFieldDataCache.java:122)  
> > > > at org.elasticsearch.search.facet.terms.strings.**  
> > > > TermsStringOrdi **nalsFacetCollec** tor.**doSetNextReader(**  
> > > > TermsStringOrd **inalsFacetCollec** tor.java:128)  
> > > > at org.elasticsearch.search. **facet**.AbstractFacetCollector.\*\*  
> > > > setNex**tReader(AbstractFacetCollector.java:75)  
> > > > at org.elasticsearch.common.lucene.MultiCollector.  
> > > > setNextReader**(MultiCollector.**java:67)  
> > > > at org.apache.lucene.search.IndexSearcher.search(**  
> > > > IndexSearcher._java:576)  
> > > > at org.elasticsearch.search. **inter** nal.ContextIndexSearcher.  
> > > > \*searc__h(ContextIndexSearcher.**java:195)  
> > > > at org.apache.lucene.search.IndexSearcher.search(  
> > > > IndexSearcher.java:445)  
> > > > at org.apache.lucene.search.IndexSearcher.search(  
> > > > IndexSearcher.java:426)  
> > > > at org.apache.lucene.search.IndexSearcher.search(  
> > > > IndexSearcher.java:342)  
> > > > at org.apache.lucene.search.IndexSearcher.search(  
> > > > IndexSearcher.java:330)  
> > > > at org.elasticsearch.search.query.QueryPhase.execute(  
> > > > QueryPhase**.java:194)  
> > > > at org.elasticsearch.search. **Searc** hService._\*  
> > > > executeQueryPhase( **Sea** rchService.java:234)  
> > > > at org.elasticsearch.search. **actio** n.\*\*  
> > > > SearchServiceTransportAction\*\*.**sendExecuteQuery(SearchServic  
> > > > eTransportAction.java:140)  
> > > > at org.elasticsearch.action.search.type.  
> > > > TransportSearchQueryThe **nFetchA** ction$AsyncAction.sendE  
> > > > xecuteFirstPhase(TransportSearchQueryThenFetchA **ction.java:** 80)  
> > > > at org.elasticsearch.action.search.type.**  
> > > > TransportSearchTypeActi**on$BaseAsyncAction.performFirstPhase(  
> > > > TransportSearchTypeActi **on.** java:204)  
> > > > at org.elasticsearch.action.search.type.**  
> > > > TransportSearchTypeActi**on$BaseAsyncAction.performFirstPhase(  
> > > > TransportSearchTypeActi **on.** java:191)  
> > > > at org.elasticsearch.action.search.type.**  
> > > > TransportSearchTypeActi\*\*on$\*\*BaseAsyncAction$2.run( **Trans**  
> > > > portSearchTypeAction.**java:177)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.**  
> > > > runTask(**ThreadPoolExecutor.java:886)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(  
> > > > ThreadPoo**lExecutor.java:908)  
> > > > at java.lang.Thread.run(Thread. **ja** va:662)  
> > > > [2012-05-07 02:00:20,400][INFO][monitor.jvm] [Cloak]  
> > > > [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total  
> > > > [717ms]/[2.1m], memory [7.8gb]-\>[7.6gb]/[7.9gb]
> > > > 
> > > > [2012-05-07 08:37:03,087][WARN][transport] [Cloak]  
> > > > Received response for a request that has timed out, sent [31974ms] ago,  
> > > > timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][  
> > > > **tlPRmwDxQ0ma7PJJPD2o** Xw][inet[/**172.16.144.203:9300]**]], id  
> > > > [152791]

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:28am UTC](https://discuss.elastic.co/t/jvm-exception-when-searching-on-huge-data/7579/13 "2017-07-06T03:28:20Z")

</div>