# Keep get OOM when reindex **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2020, 2:03pm UTC](https://discuss.elastic.co/t/keep-get-oom-when-reindex/231741 "2020-05-08T14:03:57Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![taoliu2019](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@taoliu2019](https://discuss.elastic.co/u/taoliu2019)\ **Post date:** [May 8, 2020, 2:03pm UTC](https://discuss.elastic.co/t/keep-get-oom-when-reindex/231741/1 "2020-05-08T14:03:58Z")
Hi, I'm keeping getting OOM alert when try to reindex two index. to avoid make reindex impact, I already set the requests\_per\_second=5. Currently, we have a 5 node cluster. each node with 16G memory and 8 Core. Here is the reindex command that I'm using. ``` POST _reindex?wait_for_completion=false&requests_per_second=5&slices=5 { "source": { "index": "mio-company-test" }, "dest": { "index": "mio-company2" } } ``` The Error I get from the es is show below. ``` [2020-05-08T13:37:08,740][WARN][o.e.t.n.Netty4Transport] [kt_ySjr] exception caught on transport layer [NettyTcpChannel{localAddress=/10.0.2.132:53796, remoteAddress=10.0.2.89/10.0.2.89:9300}], closing connection io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Java heap space at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:269) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:844) [?:?] Caused by: java.lang.OutOfMemoryError: Java heap space [2020-05-08T13:37:08,674][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [Thread-45], exiting java.lang.OutOfMemoryError: Java heap space at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:200) ~[?:?] at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:676) ~[?:?] at io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(PoolArena.java:691) ~[?:?] at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:260) ~[?:?] at io.netty.buffer.PoolArena.allocate(PoolArena.java:231) ~[?:?] at io.netty.buffer.PoolArena.reallocate(PoolArena.java:397) ~[?:?] at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) ~[?:?] at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:285) ~[?:?] at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:265) ~[?:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1077) ~[?:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1070) ~[?:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1060) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?] at java.lang.Thread.run(Thread.java:844) [?:?] [2020-05-08T13:37:08,727][WARN][o.e.t.n.Netty4Transport] [kt_ySjr] exception caught on transport layer [NettyTcpChannel{localAddress=/10.0.2.132:53794, remoteAddress=10.0.2.89/10.0.2.89:9300}], closing connection io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Java heap space at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:269) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:844) [?:?] Caused by: java.lang.OutOfMemoryError: Java heap space at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:200) ~[?:?] at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:676) ~[?:?] at io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(PoolArena.java:691) ~[?:?] at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:260) ~[?:?] at io.netty.buffer.PoolArena.allocate(PoolArena.java:231) ~[?:?] at io.netty.buffer.PoolArena.reallocate(PoolArena.java:397) ~[?:?] at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) ~[?:?] at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:285) ~[?:?] at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:265) ~[?:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1077) ~[?:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1070) ~[?:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1060) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263) ~[?:?] ... 18 more ``` any ideas about how to solve the issue? Upgrade all the instance to 32GB might be the last step, I still don't get why reindex would take so much memory and it can cause OOM warnings. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 10, 2020, 11:37pm UTC](https://discuss.elastic.co/t/keep-get-oom-when-reindex/231741/2 "2020-05-10T23:37:23Z")
> [@taoliu2019](#): > > requests\_per\_second=5&slices=5 Is there a specific reason you are defining these and not just using the defaults? ---
**Author:** ![taoliu2019](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@taoliu2019](https://discuss.elastic.co/u/taoliu2019)\ **Post date:** [May 11, 2020, 8:17am UTC](https://discuss.elastic.co/t/keep-get-oom-when-reindex/231741/3 "2020-05-11T08:17:27Z")
default will oom as well, define requrests\_per\_second just want to limit the impact. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2020, 8:17am UTC](https://discuss.elastic.co/t/keep-get-oom-when-reindex/231741/4 "2020-06-08T08:17:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.