ElasticSearch 5.1.1 - java.lang.OutOfMemoryError: Java heap space - Ubuntu 16.04

tomekit · January 13, 2017, 2:34pm

Hi,
I've experienced this error during heavy bulk index.
My node is 4GB RAM, 2 vCPUs.
Heap size is set to 1GB (/etc/elasticsearch/jvm.options):
-Xms1g
-Xmx1g

bootstrap.memory_lock is not set to true. However I am not using SWAP on that node.
OS is Ubuntu 16.04.

Any ideas?

This is the error:

[2017-01-13T10:47:09,831][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v2-es5-eu] fatal error in thread [Thread-18046], exiting
java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:656) ~[?:?]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[?:?]
at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:247) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133) ~[?:?]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[?:?]

dadoonet · January 13, 2017, 3:00pm

Add more RAM if you are doing "heavy" things.
2Gb of heap which is the default should help.

jprante · January 13, 2017, 4:02pm

Try ES 5.1.2 because of https://github.com/elastic/elasticsearch/pull/22452 or add system property -Dio.netty.recycler.maxCapacityPerThread=0 in 5.1.1

tomekit · January 13, 2017, 6:15pm

Right, so with 2GB of Heap it has failed once again. I will give the ES 5.1.2 a try or will add the property. Thanks

[> 2017-01-13T16:07:17,116][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v2-es5-eu] fatal error in thread [Thread-182], exiting

java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:656) ~[?:?]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[?:?]
at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:247) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133) ~[?:?]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[?:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2017-01-13T16:07:17,161][WARN ][o.e.h.n.Netty4HttpServerTransport] [v2-es5-eu] caught exception while handling client http traffic, closing connection [id: 0xa7fac867, L:/10.24.0.148:9200 - R:/10.24.0.103:57308]
java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:656) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:247) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133) ~[netty-buffer-4.1.6.Final.jar:4.1.6.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-common-4.1.6.Final.jar:4.1.6.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

jasontedor · January 13, 2017, 7:21pm

Disabling the recycler will definitely help you here.

tomekit · January 16, 2017, 8:10am

I've upgraded ES to 5.1.2 and it looks like problem has gone.
Thank you all for help.

jasontedor · January 16, 2017, 1:24pm

Yes, the recycler is now disabled by default starting in 5.1.2.

system · February 13, 2017, 1:24pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.