Seeing java.lang.OutOfMemoryError after upgrade from 0.17.4 to 0.17.8 in client

danpolites · October 25, 2011, 4:08pm

We have a client node running in a Grails application that is hosted
in a Tomcat instance. We have recently started seeing this error after
the upgrade to 0.17.8. Thoughts?

Here is the JAVA_OPTS configuration that we have been using. This
hasn't changed in awhile.

export JAVA_OPTS="-jvm server -XX:+DisableExplicitGC -XX:
+UseConcMarkSweepGC -XX:ParallelGCThreads=20 -XX:+UseParNewGC -
XX:MaxPermSize=160m -XX:PermSize=160m -XX:SurvivorRatio=8 -
XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -Xms2048m -
Xmx2048m"

Here is the error message that we get in the Tomcat logs.

Exception in thread "elasticsearch[cached]-pool-1-thread-32"
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.compress.lzf.BufferRecycler.allocEncodingBuffer(BufferRecycler.java:
80)
at
org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java:
70)
at
org.elasticsearch.common.io.stream.LZFStreamOutput.(LZFStreamOutput.java:
47)
at
org.elasticsearch.common.io.stream.CachedStreamOutput.newEntry(CachedStreamOutput.java:
37)
at
org.elasticsearch.common.io.stream.CachedStreamOutput.popEntry(CachedStreamOutput.java:
114)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
457)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
178)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
165)
at
org.elasticsearch.discovery.zen.membership.MembershipAction.sendJoinRequest(MembershipAction.java:
80)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MasterNodeFailureListener.onDisconnectedFromMaster(ZenDiscovery.java:
638)
at org.elasticsearch.discovery.zen.fd.MasterFaultDetection
$1.run(MasterFaultDetection.java:226)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "elasticsearch[cached]-pool-1-thread-38"
java.lang.OutOfMemoryError: Java heap space

kimchy · October 25, 2011, 4:58pm

Do you have a heap dump somewhere, so I can look at?

On Tue, Oct 25, 2011 at 6:08 PM, danpolites dpolites@gmail.com wrote:

We have a client node running in a Grails application that is hosted
in a Tomcat instance. We have recently started seeing this error after
the upgrade to 0.17.8. Thoughts?

Here is the JAVA_OPTS configuration that we have been using. This
hasn't changed in awhile.

export JAVA_OPTS="-jvm server -XX:+DisableExplicitGC -XX:
+UseConcMarkSweepGC -XX:ParallelGCThreads=20 -XX:+UseParNewGC -
XX:MaxPermSize=160m -XX:PermSize=160m -XX:SurvivorRatio=8 -
XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -Xms2048m -
Xmx2048m"

Here is the error message that we get in the Tomcat logs.

Exception in thread "elasticsearch[cached]-pool-1-thread-32"
java.lang.OutOfMemoryError: Java heap space
at

org.elasticsearch.common.compress.lzf.BufferRecycler.allocEncodingBuffer(BufferRecycler.java:
80)
at

org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java:
70)
at

org.elasticsearch.common.io.stream.LZFStreamOutput.(LZFStreamOutput.java:
47)
at

org.elasticsearch.common.io.stream.CachedStreamOutput.newEntry(CachedStreamOutput.java:
37)
at

org.elasticsearch.common.io.stream.CachedStreamOutput.popEntry(CachedStreamOutput.java:
114)
at

org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
457)
at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
178)
at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
165)
at

org.elasticsearch.discovery.zen.membership.MembershipAction.sendJoinRequest(MembershipAction.java:
80)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MasterNodeFailureListener.onDisconnectedFromMaster(ZenDiscovery.java:
638)
at org.elasticsearch.discovery.zen.fd.MasterFaultDetection
$1.run(MasterFaultDetection.java:226)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "elasticsearch[cached]-pool-1-thread-38"
java.lang.OutOfMemoryError: Java heap space

danpolites · October 26, 2011, 3:21am

I do not. If I see it again, I'll get one before I reboot.

On Oct 25, 12:58 pm, Shay Banon kim...@gmail.com wrote:

Do you have a heap dump somewhere, so I can look at?

On Tue, Oct 25, 2011 at 6:08 PM, danpolites dpoli...@gmail.com wrote:

We have a client node running in a Grails application that is hosted
in a Tomcat instance. We have recently started seeing this error after
the upgrade to 0.17.8. Thoughts?

Here is the JAVA_OPTS configuration that we have been using. This
hasn't changed in awhile.

export JAVA_OPTS="-jvm server -XX:+DisableExplicitGC -XX:
+UseConcMarkSweepGC -XX:ParallelGCThreads=20 -XX:+UseParNewGC -
XX:MaxPermSize=160m -XX:PermSize=160m -XX:SurvivorRatio=8 -
XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -Xms2048m -
Xmx2048m"

Here is the error message that we get in the Tomcat logs.

Exception in thread "elasticsearch[cached]-pool-1-thread-32"
java.lang.OutOfMemoryError: Java heap space
at

org.elasticsearch.common.compress.lzf.BufferRecycler.allocEncodingBuffer(Bu fferRecycler.java:
80)
at

org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java :
70)
at

org.elasticsearch.common.io.stream.LZFStreamOutput.(LZFStreamOutput.j ava:
47)
at

org.elasticsearch.common.io.stream.CachedStreamOutput.newEntry(CachedStream Output.java:
37)
at

org.elasticsearch.common.io.stream.CachedStreamOutput.popEntry(CachedStream Output.java:
114)
at

org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport .java:
457)
at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
178)
at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
165)
at

org.elasticsearch.discovery.zen.membership.MembershipAction.sendJoinRequest (MembershipAction.java:
80)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MasterNodeFailureListener.onDisconnectedFromMaster(ZenDiscovery.java:
638)
at org.elasticsearch.discovery.zen.fd.MasterFaultDetection
$1.run(MasterFaultDetection.java:226)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "elasticsearch[cached]-pool-1-thread-38"
java.lang.OutOfMemoryError: Java heap space

Thomas_Peuss · October 26, 2011, 9:12am

You can set the JVM parameter -XX:-HeapDumpOnOutOfMemoryError to get
an automatic heap dump. With -XX:HeapDumpPath=... you can control
where it gets written.

On 26 Okt., 05:21, danpolites dpoli...@gmail.com wrote:

I do not. If I see it again, I'll get one before I reboot.

On Oct 25, 12:58 pm, Shay Banon kim...@gmail.com wrote:

Do you have a heap dump somewhere, so I can look at?

On Tue, Oct 25, 2011 at 6:08 PM, danpolites dpoli...@gmail.com wrote:

We have a client node running in a Grails application that is hosted
in a Tomcat instance. We have recently started seeing this error after
the upgrade to 0.17.8. Thoughts?

Here is the JAVA_OPTS configuration that we have been using. This
hasn't changed in awhile.

export JAVA_OPTS="-jvm server -XX:+DisableExplicitGC -XX:
+UseConcMarkSweepGC -XX:ParallelGCThreads=20 -XX:+UseParNewGC -
XX:MaxPermSize=160m -XX:PermSize=160m -XX:SurvivorRatio=8 -
XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -Xms2048m -
Xmx2048m"

Here is the error message that we get in the Tomcat logs.

Exception in thread "elasticsearch[cached]-pool-1-thread-32"
java.lang.OutOfMemoryError: Java heap space
at

org.elasticsearch.common.compress.lzf.BufferRecycler.allocEncodingBuffer(Bu fferRecycler.java:
80)
at

org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java :
70)
at

org.elasticsearch.common.io.stream.LZFStreamOutput.(LZFStreamOutput.j ava:
47)
at

org.elasticsearch.common.io.stream.CachedStreamOutput.newEntry(CachedStream Output.java:
37)
at

org.elasticsearch.common.io.stream.CachedStreamOutput.popEntry(CachedStream Output.java:
114)
at

org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport .java:
457)
at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
178)
at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
165)
at

org.elasticsearch.discovery.zen.membership.MembershipAction.sendJoinRequest (MembershipAction.java:
80)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MasterNodeFailureListener.onDisconnectedFromMaster(ZenDiscovery.java:
638)
at org.elasticsearch.discovery.zen.fd.MasterFaultDetection
$1.run(MasterFaultDetection.java:226)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "elasticsearch[cached]-pool-1-thread-38"
java.lang.OutOfMemoryError: Java heap space

ABC · January 7, 2013, 2:47pm

Guys,
Was this issue resolved in the end? I have started getting the same
exception in the logs. any known solutions on solaris?
thanks!

On Wednesday, 26 October 2011 10:12:02 UTC+1, Thomas Peuss wrote:

You can set the JVM parameter -XX:-HeapDumpOnOutOfMemoryError to get
an automatic heap dump. With -XX:HeapDumpPath=... you can control
where it gets written.

On 26 Okt., 05:21, danpolites dpoli...@gmail.com wrote:

I do not. If I see it again, I'll get one before I reboot.

On Oct 25, 12:58 pm, Shay Banon kim...@gmail.com wrote:

Do you have a heap dump somewhere, so I can look at?

On Tue, Oct 25, 2011 at 6:08 PM, danpolites dpoli...@gmail.com
wrote:

We have a client node running in a Grails application that is hosted
in a Tomcat instance. We have recently started seeing this error
after
the upgrade to 0.17.8. Thoughts?

Here is the JAVA_OPTS configuration that we have been using. This
hasn't changed in awhile.

export JAVA_OPTS="-jvm server -XX:+DisableExplicitGC -XX:
+UseConcMarkSweepGC -XX:ParallelGCThreads=20 -XX:+UseParNewGC -
XX:MaxPermSize=160m -XX:PermSize=160m -XX:SurvivorRatio=8 -
XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -Xms2048m -
Xmx2048m"

Here is the error message that we get in the Tomcat logs.

Exception in thread "elasticsearch[cached]-pool-1-thread-32"
java.lang.OutOfMemoryError: Java heap space
at

org.elasticsearch.common.compress.lzf.BufferRecycler.allocEncodingBuffer(Bu
fferRecycler.java:

at

org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java
:

at

org.elasticsearch.common.io.stream.LZFStreamOutput.(LZFStreamOutput.j
ava:

at

org.elasticsearch.common.io.stream.CachedStreamOutput.newEntry(CachedStream
Output.java:

at

org.elasticsearch.common.io.stream.CachedStreamOutput.popEntry(CachedStream
Output.java:

at

org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport
.java:

at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.j
ava:

at

org.elasticsearch.transport.TransportService.sendRequest(TransportService.j
ava:

at

org.elasticsearch.discovery.zen.membership.MembershipAction.sendJoinRequest
(MembershipAction.java:

at org.elasticsearch.discovery.zen.ZenDiscovery

$MasterNodeFailureListener.onDisconnectedFromMaster(ZenDiscovery.java:

at org.elasticsearch.discovery.zen.fd.MasterFaultDetection
$1.run(MasterFaultDetection.java:226)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "elasticsearch[cached]-pool-1-thread-38"
java.lang.OutOfMemoryError: Java heap space

--