[OutOfMemoryError[Java heap space]]

Elastic Stack Elasticsearch
1

I'm getting lots of errors in my logs today with messages like:

[OutOfMemoryError[Java heap space]]

And:

org.elasticsearch.transport.NodeNotConnectedException

I'm guessing the NodeNotConnectedException is probably because the processes are crashing with OOMs?

How do I debug an OutOfMemoryError? Is this ever an expected condition in ES? E.g. because I'm running with too much data compared to the amount of RAM that I have? Or is this always indicative of a bug in ES and perhaps there'd be a chance it would be solved if I upgraded to a newer version? (I'm currently running 1.4.0) Or could this be because I'm running with a river and some of them are unstable? Or are certain queries being run consuming inordinate amounts of memory? I just have no idea where to begin to track this down.

Thanks,
Ben

2

OOM means you have too much data for the node. And the node disconnect will be related.

High level, your options are;

  1. Close or delete some indices
  2. Add more heap to the existing nodes
  3. Add more nodes to handle the data.

Upgrading can help, but you will likely run into this again. Rivers have been known to cause issues.
Monitoring, using Marvel, or similar, is important to prevent this and also to watch general resource consumption.