Hello,
I am facing java.lang.OutOfMemoryError: Java heap space, errors resulting in cluster crash on ES 5.2.0, below is the log generated from one of the nodes in the cluster.
This is a 5 node cluster with minimum 2 masters,
All we are attempting to do is index 3 million docs (including text files) using the ingest attachment processor, which is resulting the above error.
Are there any known issues for this version
[2017-05-26T07:04:25,470][INFO ][o.e.n.Node ] initializing ...
[2017-05-26T07:04:25,548][INFO ][o.e.e.NodeEnvironment ] [GJGT9UH] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [302.9gb], net total_space [567.4gb], spins? [unknown], types [rootfs]
[2017-05-26T07:04:25,548][INFO ][o.e.e.NodeEnvironment ] [GJGT9UH] heap size [39.9gb], compressed ordinary object pointers [false]
[2017-05-26T07:04:25,617][INFO ][o.e.n.Node ] node name [GJGT9UH] derived from node ID [GJGT9UH-SFSdeZJjkPlnMQ]; set [node.name] to override
[2017-05-26T07:04:25,619][INFO ][o.e.n.Node ] version[5.2.0], pid[21768], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [aggs-matrix-stats]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [ingest-common]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [lang-expression]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [lang-groovy]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [lang-mustache]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [lang-painless]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [percolator]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [reindex]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [transport-netty3]
[2017-05-26T07:04:26,417][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded module [transport-netty4]
[2017-05-26T07:04:26,418][INFO ][o.e.p.PluginsService ] [GJGT9UH] loaded plugin [ingest-attachment]
[2017-05-26T07:04:29,801][INFO ][o.e.n.Node ] initialized
[2017-05-26T07:04:29,801][INFO ][o.e.n.Node ] [GJGT9UH] starting ...
[2017-05-26T07:04:29,916][INFO ][o.e.t.TransportService ] [GJGT9UH] publish_address {:9300}, bound_addresses {:9300}
[2017-05-26T07:04:29,920][INFO ][o.e.b.BootstrapChecks ] [GJGT9UH] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-05-26T07:04:35,156][INFO ][o.e.c.s.ClusterService ] [GJGT9UH] detected_master {AEzqo8J}{AEzqo8J3RHiFcvwiKnwYSg}{PEce-l0rQFKocM8vqIDbWg}{}{:9300}, added {{786uRr9}{786uRr9CRNWTsnfC-2DS6Q}{dbtLQTXVSXu-Y1wxAXVgvw}{}{:9300},{AEzqo8J}{AEzqo8J3RHiFcvwiKnwYSg}{PEce-l0rQFKocM8vqIDbWg}{}{:9300},{uh1Haz-}{uh1Haz-5RUOVclHX-bBtQw}{upM2ifVDSaKZEd0UUr0lnA}{}{:9300},}, reason: zen-disco-receive(from master [master {AEzqo8J}{AEzqo8J3RHiFcvwiKnwYSg}{PEce-l0rQFKocM8vqIDbWg}{}{:9300} committed version [9]])
[2017-05-26T07:04:35,248][INFO ][o.e.h.HttpServer ] [GJGT9UH] publish_address {:9200}, bound_addresses {:9200}
[2017-05-26T07:04:35,248][INFO ][o.e.n.Node ] [GJGT9UH] started
[2017-05-26T07:04:44,530][INFO ][o.e.c.s.ClusterService ] [GJGT9UH] added {{mVQ5CKg}{mVQ5CKgpRD64SZ8o9W6FwQ}{FfJrK3WpSVSReW8W1pPwsQ}{10.224.59.211}{10.224.59.211:9300},}, reason: zen-disco-receive(from master [master {AEzqo8J}{AEzqo8J3RHiFcvwiKnwYSg}{PEce-l0rQFKocM8vqIDbWg}{}{:9300} committed version [16]])
[2017-05-26T07:04:44,866][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][15] overhead, spent [298ms] collecting in the last [1s]
[2017-05-26T07:04:45,867][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][16] overhead, spent [263ms] collecting in the last [1s]
[2017-05-26T07:04:59,019][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][29] overhead, spent [303ms] collecting in the last [1s]collecting in the last [1s]
[2017-05-26T07:05:22,348][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][52] overhead, spent [289ms]
[2017-05-26T07:15:42,389][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][669] overhead, spent [252ms] collecting in the last [1s]
[2017-05-26T07:45:06,428][DEBUG][o.e.a.a.c.n.i.TransportNodesInfoAction] [GJGT9UH] failed to execute on node [786uRr9CRNWTsnfC-2DS6Q]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [786uRr9][:9300][cluster:monitor/nodes/info[n]] request_id [250660] timed out after [2000ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:908) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-05-26T07:45:06,434][WARN ][o.e.a.a.c.n.i.TransportNodesInfoAction] [GJGT9UH] not accumulating exceptions, excluding exception from response
org.elasticsearch.action.FailedNodeException: Failed node [786uRr9CRNWTsnfC-2DS6Q]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:247) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$300(TransportNodesAction.java:160) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:219) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1024) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:907) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [786uRr9][:9300][cluster:monitor/nodes/info[n]] request_id [250660] timed out after [2000ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:908) ~[elasticsearch-5.2.0.jar:5.2.0]
... 4 more
[2017-05-26T07:45:07,615][WARN ][o.e.t.TransportService ] [GJGT9UH] Received response for a request that has timed out, sent [3189ms] ago, timed out [1189ms] ago, action [cluster:monitor/nodes/info[n]], node [{786uRr9}{786uRr9CRNWTsnfC-2DS6Q}{dbtLQTXVSXu-Y1wxAXVgvw}{}{:9300}], id [250660]
[2017-05-26T07:45:15,255][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][2435] overhead, spent [278ms] collecting in the last [1s]
[2017-05-26T07:45:16,255][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][2436] overhead, spent [420ms] collecting in the last [1s]
[2017-05-26T07:45:17,320][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][2437] overhead, spent [431ms] collecting in the last [1s]
[2017-05-26T07:45:21,320][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][2441] overhead, spent [275ms] collecting in the last [1s]
[2017-05-26T07:45:40,378][INFO ][o.e.m.j.JvmGcMonitorService] [GJGT9UH] [gc][2460] overhead, spent [297ms] collecting in the last [1s]collecting in the last [1s]
[2017-05-26T07:54:11,830][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer
at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:140)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:77)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:851)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
Are you really running a 40GB heap?
What is your average document size? Are you using bulk requests? If so, how large are they?
Yes, we had initially set it to 30g, & faced this error, post which we increased the heap but are still facing this.
Increasing it isn't likely to help.
Do you have the Monitoring plugin installed? If not, how are you monitoring things?
Also there's no OOM error in those logs, can you show one?
[2017-05-26T08:41:09,809][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[GJGT9UH][index][T#5]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664) ~[?:1.8.0_121]
at java.lang.String.(String.java:207) ~[?:1.8.0_121]
at java.lang.StringBuilder.toString(StringBuilder.java:407) ~[?:1.8.0_121]
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:356) ~[jackson-core-2.8.6.jar:2.8.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2470) ~[jackson-core-2.8.6.jar:2.8.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.6.jar:2.8.6]
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:86) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:352) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:300) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:263) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:218) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:105) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:78) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.index.IndexRequest.sourceAsMap(IndexRequest.java:410) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.ingest.PipelineExecutionService.innerExecute(PipelineExecutionService.java:164) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.ingest.PipelineExecutionService.access$000(PipelineExecutionService.java:41) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.ingest.PipelineExecutionService$1.doRun(PipelineExecutionService.java:65) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
I don't have any monitoring plugin installed ATM ,i've been monitoring the cluster using cerebro
Oh 5.2!
Upgrade to 5.2.2, there's known issues with heap leaks in 5.2.0.
Oh, is it, didn't see any known issues / reported issues for 5.2.0 earlier, I will surely upgrade to 5.2.2 & observe the cluster.
In the mean time, could you let me know apart from the release notes for 5.2.2 are there any API / rest level changes as I am using NEST client to connect with the elastic cluster, so just an upgrade of the version would work for me or should i be expecting any code level changes from my application side.
Thanks for your prompt responses.
No there won't be, it's a bug fix only.
Is it safe to go for 5.2.2, i could see similar issue reported for 5.2.2 ES 5.2.2: Sudden heap spikes followed by cluster crash
Try going to 5.3.2 then 
Hello @warkolm,
Upgraded to 5.3.2 & still facing the same issue.
Are there any specific settings I am missing out.
[2017-05-29T07:45:19,528][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [Thread-22], exiting
java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(PoolArena.java:661) ~[?:?]
at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:246) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:224) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[?:?]
at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:272) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[?:?]
at io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1653) ~[?:?]
at io.netty.buffer.CompositeByteBuf.consolidateIfNeeded(CompositeByteBuf.java:405) ~[?:?]
at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:196) ~[?:?]
at io.netty.handler.codec.MessageAggregator.appendPartialContent(MessageAggregator.java:317) ~[?:?]
at io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:282) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) ~[?:?]
still receiving this
[2017-05-30T20:51:12,241][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:140) at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:81) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:851) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:63) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:745)
[2017-05-30T20:51:12,245][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [Thread-7], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664) ~[?:1.8.0_121]
at java.lang.String.<init>(String.java:207) ~[?:1.8.0_121]
at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:567) ~[?:1.8.0_121]
at java.nio.CharBuffer.toString(CharBuffer.java:1241) ~[?:1.8.0_121]
at io.netty.buffer.ByteBufUtil.decodeString(ByteBufUtil.java:628) ~[?:?]
at io.netty.buffer.AbstractByteBuf.toString(AbstractByteBuf.java:1213) ~[?:?]
at org.elasticsearch.transport.netty4.ByteBufBytesReference.utf8ToString(ByteBufBytesReference.java:74) ~[?:?]
at org.elasticsearch.common.xcontent.XContentHelper.convertToJson(XContentHelper.java:165) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.XContentHelper.convertToJson(XContentHelper.java:154) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.XContentHelper.convertToJson(XContentHelper.java:149) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.action.index.IndexRequest.toString(IndexRequest.java:698) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.action.support.replication.ReplicationRequest.getDescription(ReplicationRequest.java:249) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.action.support.replication.ReplicationRequest.createTask(ReplicationRequest.java:225) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.tasks.TaskManager.register(TaskManager.java:87) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:80) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:404) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:416) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.rest.action.document.RestIndexAction.lambda$prepareRequest$1(RestIndexAction.java:90) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.rest.action.document.RestIndexAction$$Lambda$1593/1212503415.accept(Unknown Source) ~[?:?]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:260) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:199) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:505) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:72) ~[?:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:63) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]is this related ? Disable the Netty recycler
`[2017-05-30T21:40:18,773][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[AEzqo8J][bulk][T#6]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68) ~[?:1.8.0_121]
at java.lang.StringBuilder.<init>(StringBuilder.java:101) ~[?:1.8.0_121]
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:346) ~[jackson-core-2.8.6.jar:2.8.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2470) ~[jackson-core-2.8.6.jar:2.8.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.6.jar:2.8.6]
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:86) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:369) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:317) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:280) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:235) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:141) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:114) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.action.index.IndexRequest.sourceAsMap(IndexRequest.java:346) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.ingest.PipelineExecutionService.innerExecute(PipelineExecutionService.java:164) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.ingest.PipelineExecutionService.access$000(PipelineExecutionService.java:41) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.3.2.jar:5.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.3.2.jar:5.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
`© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.