Memory issues

Elastic Stack Elasticsearch
1

Hi,

WODstack started using ElasticSearch a month or two ago but we are running
into some really strange issues with instances just spinning completely out
of control and eating all the memory. It's obviously either not configured
right or tire is doing something bad but see the accompanying picture for
some insight. I honestly have no idea where to start but feel like killing
all java processes is a terribly bad thing to have to do several times a
week.

What do we do?

https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png

https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png
https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png
https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png
https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png
https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png
https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png
https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png

2

We are using the service wrapper to start ES if that means anything.

3

How much memory are you allocating to elasticsearch? Use the node stats API or big desk plugin (Elasticsearch Platform — Find real-time answers at scale | Elastic) to see where memory is being used (check heap used, and field data memory usage).

On Sunday, February 12, 2012 at 4:56 PM, Mikael Henriksson wrote:

Hi,

WODstack started using Elasticsearch a month or two ago but we are running into some really strange issues with instances just spinning completely out of control and eating all the memory. It's obviously either not configured right or tire is doing something bad but see the accompanying picture for some insight. I honestly have no idea where to start but feel like killing all java processes is a terribly bad thing to have to do several times a week.

What do we do?

(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>
(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>
(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>
(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>
(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>
(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>
(https://lh3.googleusercontent.com/-suWVjWdCoQ8/TzfSr_MIqGI/AAAAAAAAAEk/O2EuC8BepFo/s1600/yikes.png)>

4

Maybe I am just reading something wrong because
https://img.skitch.com/20120212-8i9e42d17kh7xxcqbnh97i7sn1.png
and
https://img.skitch.com/20120212-tbu72jgfqxgstkgueu65i9jf3x.png

doesn't suggest the problem lies with elasticsearch. I've been using htop
to find memory hogs but maybe there is a better way of finding those?

5

htop shows the threads associated with the process, and I think it will show the memory allocated to the process on each thread, maybe thats what you are seeing?

On Sunday, February 12, 2012 at 8:13 PM, Mikael Henriksson wrote:

Maybe I am just reading something wrong because

and

doesn't suggest the problem lies with elasticsearch. I've been using htop to find memory hogs but maybe there is a better way of finding those?

6

Something worth mentioning, is from the first image note that the file
descriptors are configured to 1024, we have had issues with clusters fail
due to this being too low as well, doesn't have to do with memory just a
heads up on configuration.