Recovering from loop on "java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot flush"

Elastic Stack Elasticsearch
1

Howdy!

I recently had a failure in my log-indexing cluster wherein
ElasticSearch ceased reading from its RabbitMQ river, and has been
continually giving RefreshFailedExceptions (showing an
IllegalStateException, caused by an OutOfMemoryError) as the cause.

When in this state, the JVM does not shut itself down properly on
SIGTERM.

The error itself doesn't bother me nearly so much as not having an
automated means to recover from it; if I could configure RabbitMQ to
always exit in such cases, for instance, the outer process supervision
layer could restart it.

Thoughts?

2

Yea, thats sadly how Java works. There isn't a nice way to hook into the JVM and be notified when it runs out of memory (and the exception can basically happen anywhere, and might actually be trapped).

The service wrapper tries to help with that in allowing to configure a filter that will restart the Java process if OutOfMemory error appears in the logs it tracks.
On Sunday, March 13, 2011 at 11:49 AM, Charles Duffy wrote:
Howdy!

I recently had a failure in my log-indexing cluster wherein
Elasticsearch ceased reading from its RabbitMQ river, and has been
continually giving RefreshFailedExceptions (showing an
IllegalStateException, caused by an OutOfMemoryError) as the cause.

When in this state, the JVM does not shut itself down properly on
SIGTERM.

The error itself doesn't bother me nearly so much as not having an
automated means to recover from it; if I could configure RabbitMQ to
always exit in such cases, for instance, the outer process supervision
layer could restart it.

Thoughts?