Ok, maybe not miracle, but it made you look.
I'm running this version of Java:
java version "1.7.0_65" OpenJDK Runtime Environment (rhel-184.108.40.206.el6_5-x86_64 u65-b17) OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
I have 30GB heaps on 64GB servers with 16 cores, and a RAID 0 stripe across 4 4TB SATA 7200 disks.
I'm indexing a consistent 30k events/sec via bulk inserts, IOps on the disks range from 100-300.
I was having very frequent young GC collections, even when my heap was only about 30% used. They were anywhere from 1-4 seconds, and happening often enough to really affect the indexing throughput, so I started looking.
I know that in general, when it comes to heap tuning, it's better to just not. The VM does a very good job, in most cases. In my searching, I came across this blog, and tried the settings they suggest, and since I enabled them, I have not seen a young GC DEBUG/INFO/WARN in my ES logs at all. It's been just over 48 hours now.
Maybe it's too early to get this excited, but I wanted to share the settings and get some comments, and hopefully they will help someone else who might be struggling with this as well.
I should also mention that my cluster is made of 13 nodes, with 3 dedicated masters, 4 dedicated clients, and 6 dedicated data nodes.
Here are the settings I'm using:
/usr/bin/java -Xms30g -Xmx30g -Xss256k -Djava.awt.headless=true -server -XX:+UseCompressedOops -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:+DisableExplicitGC -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError (snip)
The only one I left out was the XX:G1NewSizePercent=3 parameter, because apparently it's not valid anymore, and the VM complained (still started though).
Anyway, enough rambling. Check this out and let me know what you think (yes, I know it says for hbase )
Hope it helps.