We're running ES 2.1.1 and are having trouble handling out of memory errors.
We've created a small script to execute on out of memory, which includes logger commands to ensure the script has actually executed, we've added +XX:OnOutOfMemoryError=/tmp/es_oom.sh to the JAVA_OPTS in elasitcsearch.in.sh, but when we push ES to run out of memory, the script does not execute. The script is executable and is owned by the elasticsearch user.
We've also added a -XX:HeapDumpPath with a location that has plenty of space for a heap dump, and is also writable by the elasticsearch user, but when an OutOfMemoryError occurs, a heap dump is not created. We have left -XX:+HeapDumpOnOutOfMemoryError in elasticsearch.in.sh as it was already there.
EDIT: Ok, turns out there was a problem with our mount for our heap dumps, I've got that problem solved now. The OnOutOfMemoryError issue remains though.
In both scenarios, we can see the OutOfMemoryError being thrown in the Elasticsearch logs, but neither of these JVM flags seem to make a difference to what's happened.
Can someone advise on why this might not be working, and any possible solutions to it?