AWS uses Intel E5-2680 v2 CPUs (10 core, 20 threads/vcpus) for c2.2xlarge which is declared with only 8 threads/vcpus for the product. That means IMHO, only 4 real cores are used, which is certainly bit low for G1, because G1 produces lots of CPU overhead, it scales much better the higher the core count is.
Also, 7.5 GB heap size is at the lower bottom of 6 GB, which is where G1 GC is targeted at:
Oracle only sees advantages of G1 when
- More than 50% of the Java heap is occupied with live data.
- The rate of object allocation rate or promotion varies significantly.
- The application is experiencing undesired long garbage collection or compaction pauses (longer than 0.5 to 1 second).
All of this has been improved over the years by ES, by JVM parameter tuning for heap to eliminate full CMS GC runs, and reducing live object count on heap, especially in ES 2.x
So, using G1 will lead to lower GC latency, but the price is lower throughput and lower performance .
But G1 is not responsible for any indexing performance degrading over time, as this GCViewer image shows
CPU: 2x AMD Opteron 6274 (32 cores)
RAM: 64 GB
Disk: 8x450GB SAS2 6Gb/s (RAID1+0)
OS: RHEL 7.2
JDK: Oracle Java 8u77
Command line flags: -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=17179869184 -XX:MaxGCPauseMillis=1000 -XX:MaxHeapSize=17179869184 -XX:NumberOfGCLogFiles=32 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation
My throughput numbers for a mixed workload which indexes ~98 mio documents while collecting docs from other indices by executing ~200-300 mio. term queries
10 minutes: ingest: 1936194 docs, 4.85 MB/s
20 minutes: ingest: 4423150 docs, 5.35 MB/s
30 minutes: ingest: 6708534 docs, 5.45 MB/s
1 hour: ingest: 13858395 docs, 5.63 MB/s
2 hours: ingest: 27939888 docs, 5.66 MB/s
3 hours: ingest: 41825762 docs, 5.68 MB/s
4 hours: ingest: 55708007 docs, 5.71 MB/s
5 hours: ingest: 69378607 docs, 5.69 MB/s
6 hours: ingest: 82587068 docs, 5.69 MB/s
7 hours: ingest: 96188962 docs, 5.66 MB/s
8 hours: ingest: 110111536 docs, 5.67 MB/s
9 hours: ingest: 124565124 docs, 5.69 MB/s
9 hours 30 minutes: ingest: 131873275 docs, 5.72 MB/s
(total ingest after 9h42m: 192.7 GB)
Index performance degradation strongly depends on the resources available to ES, and mostly if heap size is configured too low for the assigned workload. Then, GC will always freak out to avoid OOM at all cost, no matter if CMS or G1 or any other GC algorithm.