+XX:OnOutOfMemoryError not executing, heap dump not generating

Elastic Stack Elasticsearch
1

Hi,

We're running ES 2.1.1 and are having trouble handling out of memory errors.

We've created a small script to execute on out of memory, which includes logger commands to ensure the script has actually executed, we've added +XX:OnOutOfMemoryError=/tmp/es_oom.sh to the JAVA_OPTS in elasitcsearch.in.sh, but when we push ES to run out of memory, the script does not execute. The script is executable and is owned by the elasticsearch user.

We've also added a -XX:HeapDumpPath with a location that has plenty of space for a heap dump, and is also writable by the elasticsearch user, but when an OutOfMemoryError occurs, a heap dump is not created. We have left -XX:+HeapDumpOnOutOfMemoryError in elasticsearch.in.sh as it was already there.

EDIT: Ok, turns out there was a problem with our mount for our heap dumps, I've got that problem solved now. The OnOutOfMemoryError issue remains though.

In both scenarios, we can see the OutOfMemoryError being thrown in the Elasticsearch logs, but neither of these JVM flags seem to make a difference to what's happened.

Can someone advise on why this might not be working, and any possible solutions to it?

Thanks,
Daniel

2

You would install Marvel to give you some insight into what is causing the OOM.

3

Hi all,

I was reading https://github.com/elastic/elasticsearch/issues/7838 and saw that it was suggested to use "kill -9 %p" when elasticsearch runs out of memory. This sounds like an ok solution (assuming there is enough memory available to spawn a child process). But kill never seems to run, no matter where I put the -XX:OnOutOfMemoryError flag.

The problem seem to come down to the main elasticsearch script file (/bin/elasticsearch) on the line:

exec "$JAVA" $JAVA_OPTS $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH"
org.elasticsearch.bootstrap.Elasticsearch start "$@"

If I replace exec with eval it works, but that's not ideal. If nobody has a workaround that doesn't involve hot patching I suggest writing a bug.

Regards
Eoghan