Error on the network layer

Elastic Stack Elasticsearch
1

I have a problem with a single node elasticsearch instance on a m4.4xlarge with an EBS volume attached and heap size of 8 GB.

Every now and then my elasticsearch service stops with these errors in the logs :

	at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:138)
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:76)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:295)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:266)
.
.
.

and right after with :

java.lang.OutOfMemoryError: Java heap space
	at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:633) ~[?:?]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[?:?]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) ~[?:?]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[?:?]
.
.
.

But I'm sure it's running with 8GB heap space then technically it shouldn't be out of memory. Searching for the first network issue results in nothing ... looks like not many people had that issue.

2

What version, what OS and version, what JVM and version?

3

Hi!

Elasticsearch 5 on Amazon AMI (4.4.23-31.54.amzn1.x86_64) and java version "1.7.0_121"
OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)

I've increased the heap size to 32G.

4

According to the support matrix, Elasticsearch 5.0 requires Java 8, so you probably should upgrade that. You should also try to keep the heap size a bit below 32GB in order to be able to benefit from compressed pointers. I think 30 GB should be OK, but it may be possible to push it a bit higher.

5

Noted.

My box has 64 GB of memory and I've given Elasticsearch 32 GB of heap. This problem happened again.

ps aux | grep elasticsearch gives me :

``/usr/lib/jvm/jre-1.8.0/bin/java -Xms32g -Xmx32g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms8g -Xmx8g -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.2.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/data/elasticsearch -Edefault.path.conf=/etc/elasticsearch

6

Hi!

Elasticsearch 5 on Amazon AMI (4.4.23-31.54.amzn1.x86_64) and java version "1.7.0_121"
OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)

I've increased the heap size to 32G.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.