I have a cluster of 2 nodes, index 150GB data with SSD, every node have 32G RAM and I set heap size to 16GB for elasticsearch.
I use realtime index which means I write/search indices at the same time. I use bigdesk to monitor the JVM.
However, I find that the memory suddenly bumps from 50% heap size to 80% heap size every several minutes and then cause Full GC which will make search request hang up for 10 seconds. This happened frequently.
I have set the indices.store.throttle.max_bytes_per_sec for merge. And the index request is about 500MB per hour while query request per second is 150.