java.lang.OutOfMemoryError: Java heap space with search action

Elastic Stack Elasticsearch
1

Seeing the below stack trace in elasticsearch logs. What may be the possible cause for this :

[E] asticsearchUncaughtExceptionHandler - [UID=] - fatal error in thread [elasticsearch[node01][search][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.TextBuffer.carr(TextBuffer.java:778) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.util.TextBuffer.finishCurrentSegment(TextBuffer.java:632) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2489) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2465) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.1.jar:2.8.1]
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:347) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:295) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:258) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:213) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:81) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.lookup.SourceLookup.sourceAsMapAndType(SourceLookup.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.lookup.SourceLookup.loadSourceIfNeeded(SourceLookup.java:65) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:135) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:51) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:163) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:502) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:356) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:353) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
2018-05-10T10:13:09,790 search][T#6] [E] asticsearchUncaughtExceptionHandler - [UID=] - fatal error in thread [elasticsearch[node01][search][T#6]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:214) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:595) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.CodecReader.document(CodecReader.java:88) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:349) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:188) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:154) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:502) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:356) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:353) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

2

I think you'll have to provide more info. The stacktrace doesn't really provide much information, just that you ran out of memory.

What query was run? Does a single type of query consistently OOM the node? Was any other activity going on to use up heap space?

What version of Elasticsearch are you running?

If I had to guess, I'd say there was a very large aggregation that ran, and overflowed the heap due to too many buckets. But it's hard to say without more info.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.