Hello! We are trying to optimize our ES 5.6 cluster to use less heap memory.
We have 19Tb of data, 9bln of rows in ~560 shards on 2 warm, 4 cold data nodes, which are only rarely queried. Memory is scarce resource in our private cloud, so next measures were taken:
-- mappings optimized
-- on cold nodes, indices are force merged and than shrunk to 1 shard in index, resulting in 60-90Gb per shard
-- indexing of new data takes on place on "hot" nodes
-- cold node machines are 32G of ram, and recently (as we hit GC issues again due to constant high heap usage), nodes were reconfigured to use 20Gb of heap.
What are the risk of using 60% of RAM to Java heap? Can we move further with that? The cluster receives ~10 queries daily, and users can wait some extra time.
Perhaps there are more options? Unfortunately, upgrade to 6.8 and using frozen indices is not an option at the moment.