java.lang.OutOfMemoryError: Java heap space with search action

Seeing the below stack trace in elasticsearch logs. What may be the possible cause for this :

[E] asticsearchUncaughtExceptionHandler - [UID=] - fatal error in thread [elasticsearch[node01][search][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.TextBuffer.carr(TextBuffer.java:778) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.util.TextBuffer.finishCurrentSegment(TextBuffer.java:632) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2489) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2465) ~[jackson-core-2.8.1.jar:2.8.1]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.1.jar:2.8.1]
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:347) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:295) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:258) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:213) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:81) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.lookup.SourceLookup.sourceAsMapAndType(SourceLookup.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.lookup.SourceLookup.loadSourceIfNeeded(SourceLookup.java:65) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:135) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:51) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:163) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:502) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:356) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:353) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
2018-05-10T10:13:09,790 search][T#6] [E] asticsearchUncaughtExceptionHandler - [UID=] - fatal error in thread [elasticsearch[node01][search][T#6]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:214) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:595) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.CodecReader.document(CodecReader.java:88) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:349) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:188) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:154) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:502) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:356) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:353) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

I think you'll have to provide more info. The stacktrace doesn't really provide much information, just that you ran out of memory.

What query was run? Does a single type of query consistently OOM the node? Was any other activity going on to use up heap space?

What version of Elasticsearch are you running?

If I had to guess, I'd say there was a very large aggregation that ran, and overflowed the heap due to too many buckets. But it's hard to say without more info.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.