ES 5.6.2 OutOfMemory heap space

Elastic Stack Elasticsearch
1

It appears that I experienced the same error at the same time on 2 servers in my cluster. Previously we had similar issues with aggregations that caused a heap overflow, but I thought that had been resolved.
This looks similar to https://github.com/elastic/elasticsearch/issues/15892 and https://github.com/elastic/elasticsearch/issues/24790

It appears that on a BigArrays.resize there may be similar issues to https://github.com/elastic/elasticsearch/issues/24790

[2017-10-23T12:20:08,467][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es2] fatal error in thread [elasticsearch[es2][search][T#4]], exiting
java.lang.OutOfMemoryError: Java heap space
	at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:99) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:96) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:53) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:33) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:28) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.recycler.Recyclers$3.obtain(Recyclers.java:119) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.PageCacheRecycler.bytePage(PageCacheRecycler.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:117) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.BigByteArray.resize(BigByteArray.java:143) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.BigArrays.resizeInPlace(BigArrays.java:449) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:496) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:513) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.ensureCapacity(HyperLogLogPlusPlus.java:197) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.collect(HyperLogLogPlusPlus.java:232) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator$OrdinalsCollector.postCollect(CardinalityAggregator.java:280) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator.postCollectLastCollector(CardinalityAggregator.java:120) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator.getLeafCollector(CardinalityAggregator.java:111) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:149) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.BucketCollector$2.getLeafCollector(BucketCollector.java:75) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:148) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:41) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:121) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
	at org.apache.lucene.search.FilterCollector.getLeafCollector(FilterCollector.java:40) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
	at org.elasticsearch.search.query.CancellableCollector.getLeafCollector(CancellableCollector.java:51) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
	at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:196) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:472) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:421) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$16(IndicesService.java:1129) ~[elasticsearch-5.6.2.jar:5.6.2]
2

Hi Jeff,

Have your experienced the same in 5.6.3?

I have 5.6.2 in DEV cluster and 5.6.3 in both UAT and PROD.

My DEV cluster crashed with this error already 3 times without big load being applied. UAT and PROD are stable, but they have twice memory as DEV.

Do you know if there is any sense to upgrade DEV to 5.6.3 or it is just matter of RAM size and issue still exist in 5.6.3?

Regards,
Sergey

3

Sergey,
We haven't seen the same behavior, so it may be resolved, but our usage of the aggregations that were triggering this is also intermittent. I would suggest upgrading, as it seems more stable for us now. Obviously with the GA release of 6.0 you may want to take that into consideration, given that 6.0 will be a huge change. We are starting our prep work for it now.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.