# Heap space error **URL:** \ **Category:** Elasticsearch\ **Created:** [May 18, 2012, 5:54pm UTC](https://discuss.elastic.co/t/heap-space-error/7764 "2012-05-18T17:54:18Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [May 18, 2012, 5:54pm UTC](https://discuss.elastic.co/t/heap-space-error/7764/1 "2012-05-18T17:54:18Z")
I am getting out of heap space and invalid data length error. Is this something I need to change elasticsearch to have bigger heap? I am using the default. Hadoop client is sending bulk data of Json files which are around 200k avg size. [2012-05-18 10:53:08,592][WARN][transport.netty] [Portal] Exception caught on netty layer [[id: 0x73d9024f, /172.18.62.198:60052 =\> / 172.18.62.202:9300]] java.lang.OutOfMemoryError: Java heap space [2012-05-18 10:52:55,976][WARN][transport.netty] [Portal] Exception caught on netty layer [[id: 0x48359a33, /172.18.62.197:50372 =\> / 172.18.62.202:9300]] java.io.StreamCorruptedException: invalid data length: -1121714720 at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:131) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 20, 2012, 8:29pm UTC](https://discuss.elastic.co/t/heap-space-error/7764/2 "2012-05-20T20:29:14Z")
You get that on the elasticsearch cluster nodes? You might send a large bulk request, and you need to either increase teh memory allocated to ES (see the installation guide), and/or reduce the size of the bulk request. On Fri, May 18, 2012 at 7:54 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote: > I am getting out of heap space and invalid data length error. Is this > something I need to change elasticsearch to have bigger heap? I am using > the default. Hadoop client is sending bulk data of Json files which are > around 200k avg size. > > [2012-05-18 10:53:08,592][WARN][transport.netty] [Portal] Exception > caught on netty layer [[id: 0x73d9024f, /172.18.62.198:60052 =\> / > 172.18.62.202:9300]] > > java.lang.OutOfMemoryError: Java heap space > > [2012-05-18 10:52:55,976][WARN][transport.netty] [Portal] Exception > caught on netty layer [[id: 0x48359a33, /172.18.62.197:50372 =\> / > 172.18.62.202:9300]] > > java.io.StreamCorruptedException: invalid data length: -1121714720 > > at > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:131) > > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95) > > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792) > > at > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) > > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) > > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) > > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) > > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ---
**Author:** ![Jack\_Chen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_chen/32/2464_2.png) [@Jack\_Chen](https://discuss.elastic.co/u/Jack_Chen)\ **Post date:** [September 27, 2012, 6:47am UTC](https://discuss.elastic.co/t/heap-space-error/7764/3 "2012-09-27T06:47:44Z")
I'm seeing the same problem with 0.19.8. This is a portion of the logs from our cluster. [2012-09-27 02:32:36,085][INFO][monitor.jvm] [delayed\_job2] [gc][ParNew][382223][2591] duration [1.4s], collections [2]/[4s], total [1.4s]/[2.4m], memory [239.9mb]-\>[258mb]/[495.3mb], all\_pools {[Code Cache] [3.4mb]-\>[3.4mb]/[48mb]}{[Par Eden Space] [86.8mb]-\>[23.6kb]/[133.1mb]}{[Par Survivor Space] [16.6mb]-\>[0b]/[16.6mb]}{[CMS Old Gen] [136.3mb]-\>[258mb]/[345.6mb]}{[CMS Perm Gen] [23.9mb]-\>[23.9mb]/[82mb]} [2012-09-27 02:33:55,001][WARN][transport.netty] [delayed\_job2] Exception caught on netty layer [[id: 0x0843249a, /192.168.100.28:59408 =\> /192.168.100.224:9300]] java.lang.OutOfMemoryError: Java heap space [2012-09-27 02:33:55,002][WARN][transport] [elastic1] Received response for a request that has timed out, sent [55561ms] ago, timed out [25561ms] ago, action [discovery/zen/fd/ping], node [[delayed\_job2][5wYnXMqPSRaGxgJreCLtIA][inet[/192.168.100.28:9300]]{client=true, tag=delayed\_job2, data=false, max\_local\_storage\_nodes=1, master=false}], id [37563308] [2012-09-27 02:33:55,231][WARN ][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space at org.elasticsearch.common.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42) at org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34) at org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134) at org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:69) at org.elasticsearch.common.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:81) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) [2012-09-27 02:33:55,232][WARN][http.netty] [delayed\_job2] Caught exception while handling client http traffic, closing connection [id: 0x60613546, /0:0:0:0:0:0:0:1:58816 =\> /0:0:0:0:0:0:0:1:9200] java.lang.OutOfMemoryError: Java heap space at org.elasticsearch.common.jackson.util.BufferRecycler.balloc(BufferRecycler.java:102) at org.elasticsearch.common.jackson.util.BufferRecycler.allocByteBuffer(BufferRecycler.java:57) at org.elasticsearch.common.jackson.io.IOContext.allocWriteEncodingBuffer(IOContext.java:144) at org.elasticsearch.common.jackson.smile.SmileGenerator.(SmileGenerator.java:292) at org.elasticsearch.common.jackson.smile.SmileFactory.\_createJsonGenerator(SmileFactory.java:364) at org.elasticsearch.common.jackson.smile.SmileFactory.createJsonGenerator(SmileFactory.java:275) at org.elasticsearch.common.jackson.smile.SmileFactory.createJsonGenerator(SmileFactory.java:263) at org.elasticsearch.common.xcontent.smile.SmileXContent.createGenerator(SmileXContent.java:66) at org.elasticsearch.common.xcontent.XContentBuilder.(XContentBuilder.java:100) at org.elasticsearch.common.xcontent.XContentBuilder.(XContentBuilder.java:91) at org.elasticsearch.common.xcontent.XContentBuilder.builder(XContentBuilder.java:71) at org.elasticsearch.common.xcontent.smile.SmileXContent.contentBuilder(SmileXContent.java:39) at org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:97) at org.elasticsearch.search.builder.SearchSourceBuilder.buildAsBytesStream(SearchSourceBuilder.java:552) at org.elasticsearch.action.search.SearchRequest.extraSource(SearchRequest.java:361) at org.elasticsearch.rest.action.search.RestSearchAction.parseSearchRequest(RestSearchAction.java:131) at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:68) at org.elasticsearch.rest.RestController.executeHandler(RestController.java:159) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:120) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:82) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:255) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:111) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:420) [2012-09-27 02:33:55,251][WARN][transport.netty] [delayed\_job2] Exception caught on netty layer [[id: 0x0843249a, /192.168.100.28:59408 =\> /192.168.100.224:9300]] java.io.StreamCorruptedException: invalid data length: -1029436289 at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:139) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:101) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) We were only querying, no bulk importing at all. delayed\_job2 has the heap size set to 512mb for a client node, which should be enough, right? On Monday, 21 May 2012 06:29:14 UTC+10, kimchy wrote: > You get that on the elasticsearch cluster nodes? You might send a large > bulk request, and you need to either increase teh memory allocated to ES > (see the installation guide), and/or reduce the size of the bulk request. > > On Fri, May 18, 2012 at 7:54 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\ > > > wrote: > > > I am getting out of heap space and invalid data length error. Is this > > something I need to change elasticsearch to have bigger heap? I am using > > the default. Hadoop client is sending bulk data of Json files which are > > around 200k avg size. > > > > [2012-05-18 10:53:08,592][WARN][transport.netty] [Portal] Exception > > caught on netty layer [[id: 0x73d9024f, /172.18.62.198:60052 =\> / > > 172.18.62.202:9300]] > > > > java.lang.OutOfMemoryError: Java heap space > > > > [2012-05-18 10:52:55,976][WARN][transport.netty] [Portal] Exception > > caught on netty layer [[id: 0x48359a33, /172.18.62.197:50372 =\> / > > 172.18.62.202:9300]] > > > > java.io.StreamCorruptedException: invalid data length: -1121714720 > > > > at > > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:131) > > > > at > > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95) > > > > at > > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) > > > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792) > > > > at > > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) > > > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > > > at > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > > > > at > > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > > > > at > > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > > > > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) > > > > at > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) > > > > at > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) > > > > at > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) > > > > at > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:11am UTC](https://discuss.elastic.co/t/heap-space-error/7764/4 "2017-07-06T03:11:12Z")