ElasticSearch 2.2.0 memory dump and node goes out of cluster

Elastic Stack Elasticsearch
1

ElasticSearch version: 2.2.0

JVM version: 1.8

OS version: AWS Linux C3.4xlarge

Description of the problem including expected versus actual behavior:

After we restart ElasticSearch run fine like for 2-3 days and then suddenly one of node in cluster goes out of cluster and on node it show Elasticsearch running but leaves a memory dump on disk.
It doesn't come back to cluster unless we restart this.

Please find the memory dump location:
http://admarvel.s3.amazonaws.com/uploads/lokesh1/java_pid26509.hprof.zip

2

Are you asking us to analyse your heap dump?

3

I tried to look into dump and was not able to figure out whether the problem was with filter cache or not.
I suspected that the filter cache just keep going up regardless of boundary limits. and hence it goes out of memory and generating memory dump.
I post on git hub as a bug and i am being asked to post it here.
Can you please help here ?

4

I don't have time to do a heap dump analysis sorry, you probably won't find many here that will be able to either.

Are you running Marvel to keep track of what is happening in your cluster?