I'd like to get an opinion on current recommendations for running Elasticsearch with large heaps (64-128GB) please.
I understand ~30GB is generally recommended for most installs in order to benefit from compressed OOPs and that after crossing this threshold you would need to go above ~50GB to benefit.
I also understand that larger heaps can be more prone to longer garbage collection times. Is this still a concern in 2020 using the G1GC garbage collector?
For context, we are currently using Elasticsearch 7, 30GB heap size and the G1GC garbage collector. For the most part things are working well . We store data in daily indices with a single primary shard, usually 10-15GB in size,10+ million documents. There are intermittent problems caused by end users running heavy aggregations over historical data in Kibana, temporarily spiking the heap memory usage above the parent breaker limit, tripping circuit breakers (http request too large) and causing issues for other applications which query the cluster.
I understand we could amend breakers and configure things so that the heavy aggregations would be killed before affecting other clients, but ideally it would be best to allow the visualizations to run.
We are in a position to increase the heap memory of the nodes to somewhere in the region of 64 - 128GB and still leave at least 128GB RAM unused, free for filesystem cache.
Given that the G1GC collector is now considered stable , would there be any compelling reasons not to increase the heap ?