JVM Exception when searching on huge data

manoj_2 · May 7, 2012, 12:58pm

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array
Types. As an addition of data, we introduced Nested and Array Types in to
the existing index. This new fields carry even an array size of 100. When
I try to query now after indexing data, its bringing the below JVM memory.
Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested
and array index too). Today I started to run the app with a huge data. And
when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak]
[2012-05-03] loading field [field.field_type_and_value] caused out of
memory failure
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at
org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node
[[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

manoj_2 · May 7, 2012, 1:02pm

It gone more weird....

Now the Index status turned Yellow too...

"cluster_name" : "test",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 277,
"active_shards" : 461,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 91

On Monday, May 7, 2012 6:28:28 PM UTC+5:30, Manoj wrote:

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array
Types. As an addition of data, we introduced Nested and Array Types in to
the existing index. This new fields carry even an array size of 100. When
I try to query now after indexing data, its bringing the below JVM memory.
Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested
and array index too). Today I started to run the app with a huge data. And
when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak]
[2012-05-03] loading field [field.field_type_and_value] caused out of
memory failure
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at
org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node
[[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Rafal_Kuc_3 · May 7, 2012, 1:07pm

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048

set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)


    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)


    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)


    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)


    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)


    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)


    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)


    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)


    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)


    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)


    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)


    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)


    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)


    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


    at java.lang.Thread.run(Thread.java:662)

[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

manoj_2 · May 7, 2012, 1:15pm

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data
into field data cache. Do you use faceting or sorting ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array
Types. As an addition of data, we introduced Nested and Array Types in to
the existing index. This new fields carry even an array size of 100. When
I try to query now after indexing data, its bringing the below JVM memory.
Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested
and array index too). Today I started to run the app with a huge data. And
when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak]
[2012-05-03] loading field [field.field_type_and_value] caused out of
memory failure
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at
org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node
[[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Rafal_Kuc_3 · May 7, 2012, 1:24pm

Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing.

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048

set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)


    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)


    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)


    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)


    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)


    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)


    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)


    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)


    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)


    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)


    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)


    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)


    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)


    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


    at java.lang.Thread.run(Thread.java:662)

[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

manoj_2 · May 7, 2012, 2:27pm

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
"size" : "794.7gb",

"cache" : {
"field_evictions" : 0,
"field_size" : "6.4gb",
"field_size_in_bytes" : 6899525304,
"filter_count" : 325,
"filter_evictions" : 0,
"filter_size" : "55.7mb",
"filter_size_in_bytes" : 58422192
},

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

There was a thread recently about field data cache and its memory usage.
Basically, when you use faceting or sorting on a given field for the first
time, Elasticsearch needs to read data for that field and put it into field
data cache. That's why you didn't encounter that problem with small amount
of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration
time or change its type from resident to soft. I suggest to read the
following thread:
Redirecting to Google Groups ,
because it's more or less about the same situation you are currently
experiencing.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data
into field data cache. Do you use faceting or sorting ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array
Types. As an addition of data, we introduced Nested and Array Types in to
the existing index. This new fields carry even an array size of 100. When
I try to query now after indexing data, its bringing the below JVM memory.
Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested
and array index too). Today I started to run the app with a huge data. And
when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak]
[2012-05-03] loading field [field.field_type_and_value] caused out of
memory failure
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at
org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node
[[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Rafal_Kuc_3 · May 7, 2012, 2:51pm

Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error.

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit.

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for

index.cache.field.max_size: 10000

-FYI-

while running the Nodes stats we found the following detail

"size" : "794.7gb",

"cache" : {

      "field_evictions" : 0,


      "field_size" : "6.4gb",


      "field_size_in_bytes" : 6899525304,


      "filter_count" : 325,


      "filter_evictions" : 0,


      "filter_size" : "55.7mb",


      "filter_size_in_bytes" : 58422192


    },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing.

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048

set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)


    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)


    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)


    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)


    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)


    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)


    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)


    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)


    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)


    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)


    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)


    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)


    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)


    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


    at java.lang.Thread.run(Thread.java:662)

[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

manoj_2 · May 7, 2012, 3:45pm

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

/usr/share/es/bin/service/elasticsearch.conf
another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and
index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

Hello!

Yes, this may be the reason your cluster went from Green to Yellow state,
for example one of your nodes got disconnected from the cluster because of
the garbage collection which happened before JVM reported OutOfMemory
error.

I can't say how many entries in the field data cache will be OK for you.
You may need to experiment with that. In the stats you pasted, you can see
that while doing the stats, your cache was already at 6.4gb for that node.
Given that you have ES_MAX_MEM set to 8gb you are probably near the end of
the heap limit.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
"size" : "794.7gb",

"cache" : {
"field_evictions" : 0,
"field_size" : "6.4gb",
"field_size_in_bytes" : 6899525304,
"filter_count" : 325,
"filter_evictions" : 0,
"filter_size" : "55.7mb",
"filter_size_in_bytes" : 58422192
},

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage.
Basically, when you use faceting or sorting on a given field for the first
time, Elasticsearch needs to read data for that field and put it into field
data cache. That's why you didn't encounter that problem with small amount
of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration
time or change its type from resident to soft. I suggest to read the
following thread: https://groups.google.https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
com/forum/?fromgroups#!topic/https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
elasticsearch/DLcErfeivQkhttps://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk ,
because it's more or less about the same situation you are currently
experiencing.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data
into field data cache. Do you use faceting or sorting ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array
Types. As an addition of data, we introduced Nested and Array Types in to
the existing index. This new fields carry even an array size of 100. When
I try to query now after indexing data, its bringing the below JVM memory.
Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested
and array index too). Today I started to run the app with a huge data. And
when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak]
[2012-05-03] loading field [field.field_type_and_value] caused out of
memory failure
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at
org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node
[[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Rafal_Kuc_3 · May 7, 2012, 3:50pm

Hello!

You need to check which of those two files is the one that is actually used in your system (maybe one is a link to the other ?).

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

/usr/share/es/bin/service/elasticsearch.conf
another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error.

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit.

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for

index.cache.field.max_size: 10000

-FYI-

while running the Nodes stats we found the following detail

"size" : "794.7gb",

"cache" : {

      "field_evictions" : 0,


      "field_size" : "6.4gb",


      "field_size_in_bytes" : 6899525304,


      "filter_count" : 325,


      "filter_evictions" : 0,


      "filter_size" : "55.7mb",


      "filter_size_in_bytes" : 58422192


    },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: https://groups.google.com/forum/?fromgroups#!topic/e lasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing.

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:

Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types. As an addition of data, we introduced Nested and Array Types in to the existing index. This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048

set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure

java.lang.OutOfMemoryError: Java heap space

    at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)


    at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)


    at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)


    at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)


    at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)


    at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)


    at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)


    at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)


    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)


    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)


    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)


    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)


    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)


    at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)


    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)


    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)


    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


    at java.lang.Thread.run(Thread.java:662)

[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

kimchy · May 9, 2012, 9:04am

Its better to simply increase the memory used for the JVM, or scale out to
more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj manokrrish@gmail.com wrote:

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

/usr/share/es/bin/service/elasticsearch.conf

another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and
index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

Hello!

Yes, this may be the reason your cluster went from Green to Yellow state,
for example one of your nodes got disconnected from the cluster because of
the garbage collection which happened before JVM reported OutOfMemory
error.

I can't say how many entries in the field data cache will be OK for you.
You may need to experiment with that. In the stats you pasted, you can see
that while doing the stats, your cache was already at 6.4gb for that node.
Given that you have ES_MAX_MEM set to 8gb you are probably near the end of
the heap limit.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
"size" : "794.7gb",

"cache" : {
"field_evictions" : 0,
"field_size" : "6.4gb",
"field_size_in_bytes" : 6899525304,
"filter_count" : 325,
"filter_evictions" : 0,
"filter_size" : "55.7mb",
"filter_size_in_bytes" : 58422192
},

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage.
Basically, when you use faceting or sorting on a given field for the first
time, Elasticsearch needs to read data for that field and put it into field
data cache. That's why you didn't encounter that problem with small amount
of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration
time or change its type from resident to soft. I suggest to read the
following thread: https://groups.google.https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
com/forum/?fromgroups#!topic/https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
elasticsearch/DLcErfeivQkhttps://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk ,
because it's more or less about the same situation you are currently
experiencing.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data
into field data cache. Do you use faceting or sorting ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or
Array Types. As an addition of data, we introduced Nested and Array Types
in to the existing index. This new fields carry even an array size of 100.
When I try to query now after indexing data, its bringing the below JVM
memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of
nested and array index too). Today I started to run the app with a huge
data. And when on searching this huge index the following exception is
occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.*resident]
[Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.field.data.support.FieldDataLoader.
*load(FieldDataLoader.java:61)
at org.elasticsearch.index.field.data.strings.StringFieldData.
load(StringFieldData.java:90)
at org.elasticsearch.index.field.data.strings.
StringFieldDataType.load(StringFieldDataType.java:56)
at org.elasticsearch.index.field.data.strings.
StringFieldDataType.load(StringFieldDataType.java:34)
at org.elasticsearch.index.field.data.FieldData.load(FieldData.
java:111)
at org.elasticsearch.index.cache.field.data.support.
AbstractConcurrentMapFieldDataCache.cache(
AbstractConcurrentMapFieldDataCache.java:122)
at org.elasticsearch.search.facet.terms.strings.
TermsStringOrdinalsFacetCollector.doSetNextReader(
TermsStringOrdinalsFacetCollector.java:128)
at org.elasticsearch.search.facet.AbstractFacetCollector.
setNextReader(**AbstractFacetCollector.java:**75)
at org.elasticsearch.common.lucene.MultiCollector.
setNextReader(MultiCollector.**java:67)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:576)
at org.elasticsearch.search.internal.ContextIndexSearcher.
search(ContextIndexSearcher.**java:195)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:445)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:426)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:342)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:330)
at org.elasticsearch.search.query.QueryPhase.execute(
QueryPhase.java:194)
at org.elasticsearch.search.**SearchService.executeQueryPhase(
SearchService.java:234)
at org.elasticsearch.search.action.
SearchServiceTransportAction.sendExecuteQuery(
SearchServiceTransportAction.java:140)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.
sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$**BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.**java:204)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$**BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.**java:191)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$2.run(
TransportSearchTypeAction.**java:177)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][**
tlPRmwDxQ0ma7PJJPD2oXw][inet[/**172.16.144.203:9300]]], id [152791]

manoj_2 · May 15, 2012, 4:32am

Thanks Kimchy and Rafal....

We increased the JVM memory to 16GB. Also changed the Array type fields in
the document to String types. Achieved String type by using some
separating special characters between. Now, with this new String type data
of about 50GB, we again went for faceting. It still causing the problem...

Also, I would like to confirm, if we are using enough the machine's
efficiency by using 2 machines now, with 16GB of JVM for ES. What is the
MAX JVM I can give for the current structure?

I am just interested to know the MAX efficiency that could be pulled out of
a cluster....

Thanks a lot!

On Wednesday, May 9, 2012 2:34:09 PM UTC+5:30, kimchy wrote:

Its better to simply increase the memory used for the JVM, or scale out to
more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj manokrrish@gmail.com wrote:

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

/usr/share/es/bin/service/elasticsearch.conf

another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and
index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

Hello!

Yes, this may be the reason your cluster went from Green to Yellow
state, for example one of your nodes got disconnected from the cluster
because of the garbage collection which happened before JVM reported
OutOfMemory error.

I can't say how many entries in the field data cache will be OK for you.
You may need to experiment with that. In the stats you pasted, you can see
that while doing the stats, your cache was already at 6.4gb for that node.
Given that you have ES_MAX_MEM set to 8gb you are probably near the end of
the heap limit.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
"size" : "794.7gb",

"cache" : {
"field_evictions" : 0,
"field_size" : "6.4gb",
"field_size_in_bytes" : 6899525304,
"filter_count" : 325,
"filter_evictions" : 0,
"filter_size" : "55.7mb",
"filter_size_in_bytes" : 58422192
},

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage.
Basically, when you use faceting or sorting on a given field for the first
time, Elasticsearch needs to read data for that field and put it into field
data cache. That's why you didn't encounter that problem with small amount
of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration
time or change its type from resident to soft. I suggest to read the
following thread: https://groups.google.https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
com/forum/?fromgroups#!topic/https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
elasticsearch/DLcErfeivQkhttps://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk ,
because it's more or less about the same situation you are currently
experiencing.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field
data into field data cache. Do you use faceting or sorting ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or
Array Types. As an addition of data, we introduced Nested and Array Types
in to the existing index. This new fields carry even an array size of 100.
When I try to query now after indexing data, its bringing the below JVM
memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of
nested and array index too). Today I started to run the app with a huge
data. And when on searching this huge index the following exception is
occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.**resident]
[Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.field.data.support.FieldDataLoader.
load(FieldDataLoader.java:61)
at org.elasticsearch.index.field.data.strings.StringFieldData.
load(StringFieldData.java:90)
at org.elasticsearch.index.field.data.strings.
StringFieldDataType.load(StringFieldDataType.java:56)
at org.elasticsearch.index.field.data.strings.
StringFieldDataType.load(StringFieldDataType.java:34)
at org.elasticsearch.index.field.
data.FieldData.load(FieldData.java:111)
at org.elasticsearch.index.cache.field.data.support.
AbstractConcurrentMapFieldDataCache.cache(
AbstractConcurrentMapFieldDataCache.java:122)
at org.elasticsearch.search.facet.terms.strings.
TermsStringOrdinalsFacetCollector.doSetNextReader(
TermsStringOrdinalsFacetCollector.java:128)
at org.elasticsearch.search.facet.AbstractFacetCollector.
setNextReader(**AbstractFacetCollector.java:**75)
at org.elasticsearch.common.lucene.MultiCollector.
setNextReader(MultiCollector.**java:67)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:576)
at org.elasticsearch.search.internal.ContextIndexSearcher.
search(ContextIndexSearcher.**java:195)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:445)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:426)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:342)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:330)
at org.elasticsearch.search.query.QueryPhase.execute(
QueryPhase.java:194)
at org.elasticsearch.search.**SearchService.**executeQueryPhase(
SearchService.java:234)
at org.elasticsearch.search.action.
SearchServiceTransportAction.sendExecuteQuery(
SearchServiceTransportAction.java:140)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.
sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$**BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.**java:204)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$**BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.**java:191)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$2.run(
TransportSearchTypeAction.**java:177)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][*
*tlPRmwDxQ0ma7PJJPD2oXw][inet[/**172.16.144.203:9300]]], id [152791]

kimchy · May 17, 2012, 8:27pm

Are you asking regarding the maximum amount of memory you can allocate to
the elasticsearch java process? It shouldn't be more than the machine
memory, leaving space to the OS, but effectively, you probably want to
leave some memory as well for things like file system cache.

On Tue, May 15, 2012 at 7:32 AM, Manoj manokrrish@gmail.com wrote:

Thanks Kimchy and Rafal....

We increased the JVM memory to 16GB. Also changed the Array type fields in
the document to String types. Achieved String type by using some
separating special characters between. Now, with this new String type data
of about 50GB, we again went for faceting. It still causing the problem...

Also, I would like to confirm, if we are using enough the machine's
efficiency by using 2 machines now, with 16GB of JVM for ES. What is the
MAX JVM I can give for the current structure?

I am just interested to know the MAX efficiency that could be pulled out
of a cluster....

Thanks a lot!

On Wednesday, May 9, 2012 2:34:09 PM UTC+5:30, kimchy wrote:

Its better to simply increase the memory used for the JVM, or scale out
to more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj manokrrish@gmail.com wrote:

Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch

/usr/share/es/bin/service/**elasticsearch.conf

another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and
index.cache.field.type?

On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:

Hello!

Yes, this may be the reason your cluster went from Green to Yellow
state, for example one of your nodes got disconnected from the cluster
because of the garbage collection which happened before JVM reported
OutOfMemory error.

I can't say how many entries in the field data cache will be OK for
you. You may need to experiment with that. In the stats you pasted, you can
see that while doing the stats, your cache was already at 6.4gb for that
node. Given that you have ES_MAX_MEM set to 8gb you are probably near the
end of the heap limit.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
"size" : "794.7gb",

"cache" : {
"field_evictions" : 0,
"field_size" : "6.4gb",
"field_size_in_bytes" : 6899525304,
"filter_count" : 325,
"filter_evictions" : 0,
"filter_size" : "55.7mb",
"filter_size_in_bytes" : 58422192
},

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory
usage. Basically, when you use faceting or sorting on a given field for the
first time, Elasticsearch needs to read data for that field and put it into
field data cache. That's why you didn't encounter that problem with small
amount of data and you started seeing it after the amount of data changed.

What you can do is limit your field data cache size, set it expiration
time or change its type from resident to soft. I suggest to read the
following thread: https://groups.google.https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
com/forum/?fromgroups#!topic/https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk
elasticsearch/DLcErfeivQkhttps://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk ,
because it's more or less about the same situation you are currently
experiencing.

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field
data into field data cache. Do you use faceting or sorting ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or
Array Types. As an addition of data, we introduced Nested and Array Types
in to the existing index. This new fields carry even an array size of 100.
When I try to query now after indexing data, its bringing the below JVM
memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of
nested and array index too). Today I started to run the app with a huge
data. And when on searching this huge index the following exception is
occurring...

[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident]
[Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.field.****
data.support.FieldDataLoader.load(FieldDataLoader.java:61)
at org.elasticsearch.index.field.****
data.strings.StringFieldData.load(StringFieldData.java:90)
at org.elasticsearch.index.field.data.strings.
StringFieldDataType.load(StringFieldDataType.java:56)
at org.elasticsearch.index.field.**data.strings.
StringFieldDataType.load(StringFieldDataType.java:34)
at org.elasticsearch.index.field.
data.FieldData.load(FieldData.****java:111)
at org.elasticsearch.index.cache.**field.data.support.
AbstractConcurrentMapFieldDataCache.cache(
AbstractConcurrentMapFieldDataCache.java:122)
at org.elasticsearch.search.facet.terms.strings.
TermsStringOrdinalsFacetCollector.doSetNextReader(
TermsStringOrdinalsFacetCollector.java:128)
at org.elasticsearch.search.facet.AbstractFacetCollector.**
setNextReader(AbstractFacetCollector.java:75)
at org.elasticsearch.common.lucene.MultiCollector.
setNextReader(MultiCollector.java:67)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:576)
at org.elasticsearch.search.internal.ContextIndexSearcher.
*search(ContextIndexSearcher.java:195)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:445)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:426)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:342)
at org.apache.lucene.search.IndexSearcher.search(
IndexSearcher.java:330)
at org.elasticsearch.search.query.QueryPhase.execute(
QueryPhase.java:194)
at org.elasticsearch.search.SearchService.*
executeQueryPhase(SearchService.java:234)
at org.elasticsearch.search.action.**
SearchServiceTransportAction**.sendExecuteQuery(SearchServic
eTransportAction.java:140)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.sendE
xecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.java:204)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.java:191)
at org.elasticsearch.action.search.type.
TransportSearchTypeActi**on$**BaseAsyncAction$2.run(Trans
portSearchTypeAction.java:177)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm ] [Cloak]
[gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total
[717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport ] [Cloak]
Received response for a request that has timed out, sent [31974ms] ago,
timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][
tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id
[152791]

Topic		Replies	Views
Java Out Of Memory error while indexing data into elasticsearch Elasticsearch	4	1377	July 6, 2017
JVM Heap Size Elasticsearch	5	1355	October 19, 2017
ElasticSearch Cluster JVM Heap Increasing level while doing Queries and Writting Elasticsearch	4	833	July 1, 2019
Elastic Search-Java.lang.OutOfMemoryError Elasticsearch	2	1803	October 15, 2018
Elasticsearch (6.4.1) - JVM OutOfMemoryError Elasticsearch	5	1000	June 26, 2019

JVM Exception when searching on huge data

Related topics