I have an issue with elasticsearch and HEAP usage during snapshots.
My cluster have 7 nodes (2cpu & 7.5G of RAM per node, local SSD storage), about 220 millions of documents for a total size of 377GB worth of data (reported by Kibana).
I'm running version 5.5.2 (I Know it is EOL but that's what I'm stuck with).
I'm currently implementing hourly snapshots to a s3 bucket.
While everything is working fine, during the snapshot the JVM heap rises over 85%, triggering our monitoring system.
I have set the heap size to 50% of my available RAM according to the manual, so I think configuration is good.
I attached a capture of one node during the snapshot.
I'm strugling to find something to do to prevent such JVP HEAP spikes.
Is it dangerous (can crash the cluster/node)?
What can I do about this?
Thanks for your help