For us, these nodes are c3.2xlarge's on AWS set with a 7.5GB heap. We are still on ES 1.7.4 though, and running java 1.8.0_66. We'll be making the update to ES 2.x in a few months and will no doubt retry this whole experiment, since it was so easy to reproduce.
AWS uses Intel E5-2680 v2 CPUs (10 core, 20 threads/vcpus) for c2.2xlarge which is declared with only 8 threads/vcpus for the product. That means IMHO, only 4 real cores are used, which is certainly bit low for G1, because G1 produces lots of CPU overhead, it scales much better the higher the core count is.
Also, 7.5 GB heap size is at the lower bottom of 6 GB, which is where G1 GC is targeted at:
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/g1_gc.html
Oracle only sees advantages of G1 when
- More than 50% of the Java heap is occupied with live data.
- The rate of object allocation rate or promotion varies significantly.
- The application is experiencing undesired long garbage collection or compaction pauses (longer than 0.5 to 1 second).
All of this has been improved over the years by ES, by JVM parameter tuning for heap to eliminate full CMS GC runs, and reducing live object count on heap, especially in ES 2.x
So, using G1 will lead to lower GC latency, but the price is lower throughput and lower performance .
But G1 is not responsible for any indexing performance degrading over time, as this GCViewer image shows
CPU: 2x AMD Opteron 6274 (32 cores)
RAM: 64 GB
Disk: 8x450GB SAS2 6Gb/s (RAID1+0)
OS: RHEL 7.2
JDK: Oracle Java 8u77
Command line flags: -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=17179869184 -XX:MaxGCPauseMillis=1000 -XX:MaxHeapSize=17179869184 -XX:NumberOfGCLogFiles=32 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation
My throughput numbers for a mixed workload which indexes ~98 mio documents while collecting docs from other indices by executing ~200-300 mio. term queries
10 minutes: ingest: 1936194 docs, 4.85 MB/s
20 minutes: ingest: 4423150 docs, 5.35 MB/s
30 minutes: ingest: 6708534 docs, 5.45 MB/s
1 hour: ingest: 13858395 docs, 5.63 MB/s
2 hours: ingest: 27939888 docs, 5.66 MB/s
3 hours: ingest: 41825762 docs, 5.68 MB/s
4 hours: ingest: 55708007 docs, 5.71 MB/s
5 hours: ingest: 69378607 docs, 5.69 MB/s
6 hours: ingest: 82587068 docs, 5.69 MB/s
7 hours: ingest: 96188962 docs, 5.66 MB/s
8 hours: ingest: 110111536 docs, 5.67 MB/s
9 hours: ingest: 124565124 docs, 5.69 MB/s
9 hours 30 minutes: ingest: 131873275 docs, 5.72 MB/s
(total ingest after 9h42m: 192.7 GB)
Index performance degradation strongly depends on the resources available to ES, and mostly if heap size is configured too low for the assigned workload. Then, GC will always freak out to avoid OOM at all cost, no matter if CMS or G1 or any other GC algorithm.
Hi Jörg,
Indeed there is lots of testing and tweaking one should do with their particular usage and cluster size. All of the above information is useful for anyone else coming across this thread.
Just making sure it was clear, we never saw the indexing slowdown issue. We were seeing the other issue described in this thread which was runaway CPU problems that could only solved by restarting the troubled node. I would hazard to say that since we do not see this issue with CMS, there is still some bugs with G1GC. I know Elastic has been cautious about recommending G1 for reasons exactly like this, clearly something in our index design and/or query was tripping on a bug that is very rare. The problem is completely gone with CMS and yet 100% reproducible with G1.
For now all that matters is we have a stable ES cluster. We migrated from Solr, and those instances at the time had been up for > 2 years without any issues whatsoever. We migrated to ES and had nothing but problems. I guess we should have never enabled G1 in the first place. 
Hi Jörg,
our setup for a data-node: 2x AMD Opteron(tm) Processor 6376, 128Gbytes RAM, HP MSA 2040 (3 mount points EXT4 on RAID-6). On this server there are two ES instances (data node, 30,5Gbytes for JVM each) and one custom data loader (client node 4Gbytes RAM).
I had run G1 GC tests on my machines for weeks since end of 2011 on early JDK 8 and never saw a single crash with data loss caused by G1 GC.
Using G1 GC is not for beginners and there are some rules to follow from my own experience:
- prefer physical machines (there can be subtle memory management issues in VM guest/host implementations)
- prefer latest kernel/OS (to include all available fixes for memory management issues outside JVM)
- use always Java 8 (G1 GC on JDK7 is a nightmare, especially with the permgen space issues)
- prefer one 64bit JVM per machine for data nodes (otherwise many G1 GC instances compete against each other and steal too much CPU cycles)
- reserve at least 6 GB heap with XX:+UseG1GC (otherwise regionized heap makes not much sense and G1 is under stress with low-sized heaps)
- do not use -Xmn or any other -XX heap parameter "optimization" with -XX:+UseG1GC (or G1 GC will run crazy)
It's true that CMS GC is a very old and stable part of the JVM, where JVM engineers take care to not change code to keep stability, and G1 is relatively new where more code changes happen.
There are also other severe issues in JVM GC that may cause crashes (that can lead to data loss), so I think the statement "there are bugs in G1 GC so I can't use it" should not be overrated. Any JVM crash is serious, especially when triggered by Lucene/Elasticsearch heavy duty testing, but there are also other failing components in the JVM than just G1 GC that need to be fixed.
AFAIK ES detected G1 GC crashes against early releases of G1GC in JDK 7 and 32-bit on virtual machines which are obviously prone to crashes because a 32bit JVM has reduced address space management capabilities. These test configurations are not part of my configurations, hence I could not reproduce crashes.
Example search for open JDK issues:
1 Like
We've seen crashes on JDK 8 builds too, and there is the FP register corruption bug that impacts at least 8u60 through the latest 8u92. For further details on our current standpoint on G1 GC please see A Heap of Trouble.
Though this solved the issue of one host going hot, we had the stop the world phase for our cluster and it went red for a while, and then started the reallocation for the entire fleet.
We have a heap of 30Gb, our next step is to reduce the heap size and find out what is the impact of the same