0.19.11 StackOverflowError

Elastic Stack Elasticsearch
1

Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to
start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Log https://gist.github.com/4060559

  • David.

--

2

davrob2 wrote:

After my move to 0.19.11 - I am getting a StackOverflowError when I try to
start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Log https://gist.github.com/4060559

What version of ES were you using before? Seems like what was on
disk is not compatible with the snappy support in 0.19.11, but that's
a guess. Any changes besides ES version?

-Drew

--

3

The Snappy message is just a warning and is not related. Your stack trace
shows you have JVM stack settings that are too small for your JVM to run
ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:

Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to
start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Log https://gist.github.com/4060559

  • David.

--

4

Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what
happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty ] [Boomerang]
Message not fully read (request) for [36986] and action , resetting
[2012-11-13 09:51:09,894][WARN
][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related. I'm using the same settings I was
running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

  • David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:

The Snappy message is just a warning and is not related. Your stack trace
shows you have JVM stack settings that are too small for your JVM to run
ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:

Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try
to start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Log https://gist.github.com/4060559

  • David.

--

5

Just for the record, these are basic Java Settings:

Java home is /cs/java/jdk160_16-64b

Java bin is /cs/java/jdk160_16-64b/bin/java

JAVA_OPTS is -server -Xms3600m -Xmx3600m -Xss128k -Djline.enabled=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

CLASSPATH is
:/app/securities/rapport/deploy/dev2/ESServer/lib/elasticsearch-0.19.11.jar:/app/securities/rapport/deploy/dev2/ESServer/lib/:/app/securities/rapport/deploy/dev2/ESServer/lib/sigar/

On Tuesday, 13 November 2012 14:57:11 UTC, davrob2 wrote:

Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what
happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty ] [Boomerang]
Message not fully read (request) for [36986] and action , resetting
[2012-11-13 09:51:09,894][WARN
][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related. I'm using the same settings I was
running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

  • David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:

The Snappy message is just a warning and is not related. Your stack trace
shows you have JVM stack settings that are too small for your JVM to run
ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:

Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try
to start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Log https://gist.github.com/4060559

  • David.

--

6

You should check your custom Xss setting, 128k is too small for 64bit
stacks on Solaris, 512k is the default.

If this is Java JDK 6u16, this version is very old, released back in
October, 2009. It is not expected that you can use applications like ES
flawlessly without unexpected behavior on such old JVMs. And, many bugs and
security issues have been fixed within the last three years.

Please update to the latest JDK6 (which is 6u37) or JDK7 (the current JDK,
which is 7u9).

Jörg

On Tuesday, November 13, 2012 4:10:16 PM UTC+1, davrob2 wrote:

Just for the record, these are basic Java Settings:

Java home is /cs/java/jdk160_16-64b

Java bin is /cs/java/jdk160_16-64b/bin/java

JAVA_OPTS is -server -Xms3600m -Xmx3600m -Xss128k -Djline.enabled=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

CLASSPATH is
:/app/securities/rapport/deploy/dev2/ESServer/lib/elasticsearch-0.19.11.jar:/app/securities/rapport/deploy/dev2/ESServer/lib/:/app/securities/rapport/deploy/dev2/ESServer/lib/sigar/

On Tuesday, 13 November 2012 14:57:11 UTC, davrob2 wrote:

Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what
happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty ] [Boomerang]
Message not fully read (request) for [36986] and action , resetting
[2012-11-13 09:51:09,894][WARN
][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related. I'm using the same settings I was
running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

  • David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:

The Snappy message is just a warning and is not related. Your stack
trace shows you have JVM stack settings that are too small for your JVM to
run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:

Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try
to start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Loghttps://gist.github.com/4060559

  • David.

--

7

Thanks for the advice on the xss settings - I'll try an xss of 512k, though
I did up it to the standard in the 0.20 release without any luck:

reduce the per-thread stack size

JAVA_OPTS="$JAVA_OPTS -Xss256k"

Unfortunately, I'm probably stuck with my Java version, unless I try to get
an exception from my company, which is probably more trouble than its worth.

Luckily enough - I'll be moving my remaining Solaris environments to Linux
in the next couple of months, so I may stay with the old version of ES
until then.

On Tuesday, 13 November 2012 19:14:03 UTC, Jörg Prante wrote:

You should check your custom Xss setting, 128k is too small for 64bit
stacks on Solaris, 512k is the default.

If this is Java JDK 6u16, this version is very old, released back in
October, 2009. It is not expected that you can use applications like ES
flawlessly without unexpected behavior on such old JVMs. And, many bugs and
security issues have been fixed within the last three years.

Please update to the latest JDK6 (which is 6u37) or JDK7 (the current JDK,
which is 7u9).

Jörg

On Tuesday, November 13, 2012 4:10:16 PM UTC+1, davrob2 wrote:

Just for the record, these are basic Java Settings:

Java home is /cs/java/jdk160_16-64b

Java bin is /cs/java/jdk160_16-64b/bin/java

JAVA_OPTS is -server -Xms3600m -Xmx3600m -Xss128k -Djline.enabled=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

CLASSPATH is
:/app/securities/rapport/deploy/dev2/ESServer/lib/elasticsearch-0.19.11.jar:/app/securities/rapport/deploy/dev2/ESServer/lib/:/app/securities/rapport/deploy/dev2/ESServer/lib/sigar/

On Tuesday, 13 November 2012 14:57:11 UTC, davrob2 wrote:

Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what
happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty ] [Boomerang]
Message not fully read (request) for [36986] and action , resetting
[2012-11-13 09:51:09,894][WARN
][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related. I'm using the same settings I was
running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

  • David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:

The Snappy message is just a warning and is not related. Your stack
trace shows you have JVM stack settings that are too small for your JVM to
run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:

Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I
try to start ES, also I get a SnappyError, which may be related.

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):

0.19.11 StackOverflowError - Debug Loghttps://gist.github.com/4060559

  • David.

--

8

FYI -Xss values are only estimates, they differ between operating systems
and machine architectures, so vendors adjust the JVMs for different default
values in different environments. The ES preconfigured value is not
guaranteed to be safe. To be honest I don't know why there is
a preconfigured Xss value at all, maybe there was once assumed that thread
space consumption could be lowered because ES uses a lot of threads.

Quoting:

"You may be running into a problem with the default stack size for threads.
In Java SE 6, the default on Sparc is 512k in the 32-bit VM, and 1024k in
the 64-bit VM. On x86 Solaris/Linux it is 320k in the 32-bit VM and 1024k
in the 64-bit VM.

On Windows, the default thread stack size is read from the binary
(java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and 1024k
in the 64-bit VM.

You can reduce your stack size by running with the -Xss option. For example:

java -server -Xss64k

Note that on some versions of Windows, the OS may round up thread stack
sizes using very coarse granularity. If the requested size is less than the
default size by 1K or more, the stack size is rounded up to the default;
otherwise, the stack size is rounded up to a multiple of 1 MB.

64k is the least amount of stack space allowed per thread."
http://www.oracle.com/technetwork/java/hotspotfaq-138619.html

Jörg

--

9

Hi Jorg,

Thanks for your detailed explanation.

-David.

On Wednesday, 14 November 2012 00:18:52 UTC, Jörg Prante wrote:

FYI -Xss values are only estimates, they differ between operating systems
and machine architectures, so vendors adjust the JVMs for different default
values in different environments. The ES preconfigured value is not
guaranteed to be safe. To be honest I don't know why there is
a preconfigured Xss value at all, maybe there was once assumed that thread
space consumption could be lowered because ES uses a lot of threads.

Quoting:

"You may be running into a problem with the default stack size for
threads. In Java SE 6, the default on Sparc is 512k in the 32-bit VM, and
1024k in the 64-bit VM. On x86 Solaris/Linux it is 320k in the 32-bit VM
and 1024k in the 64-bit VM.

On Windows, the default thread stack size is read from the binary
(java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and 1024k
in the 64-bit VM.

You can reduce your stack size by running with the -Xss option. For
example:

java -server -Xss64k

Note that on some versions of Windows, the OS may round up thread stack
sizes using very coarse granularity. If the requested size is less than the
default size by 1K or more, the stack size is rounded up to the default;
otherwise, the stack size is rounded up to a multiple of 1 MB.

64k is the least amount of stack space allowed per thread."
Frequently Asked Questions About the Java HotSpot VM

Jörg

--