We are trying to evaluate an Elasticsearch upgrade from 5.6 to 7.8 and are seeing some unexpected memory pressure on our test 7.8 cluster.
Production Cluster:
Elasticsearch 5.6 - 12 Nodes - 3 Hot 9 Warm - 64GB RAM, 31GB JVM - Java 8 using CMS
Test Cluster:
Elasticsearch 7.8 - 12 Nodes - 3 Hot 9 Warm - 64GB RAM, 31GB JVM - Java 14 using G1GC
We have been running a test for a week or so to evaluate how our system handles Elasticsearch 7.8. We also monitor and log the heap usage for each node periodically.
This is a typical heap pattern on our production cluster. This graph reports the max heap.
Now this is what our test cluster looks like:
I did notice that the overall average heap usage has actually reduced. The concerning part is the spikes as we had been seeing the parent circuit breaking exception occur during these times.
To give a bit more context, from midnight to around 8am our systems do various things with the data in Elasticsearch. A BI system extracts data using scrolls and a few other programs run some aggregations. We had to tweak the aggregations because of the 10,000 bucket limit so these actually run longer with 7.8.
Our production 5.6 cluster has the same pattern of running systems extracting and aggregating but we see no spikies at all during this time.
There are no real errors that occur with our test cluster apart from increased memory spikes.
I have been checking the gc logs for any clues. Using GCeasy to generate some reports:
Test Warm Node:
Test Hot Node:
(An overactive set of aggregations causes that spike in heap after gc on the hot node)
Wondering if any GC tuning can be done to reduce the heap spikes.
GCEasy recommended increasing G1HeapRegionSize and setting UseStringDeduplication. I have done that to evaluate any improvements. Haven't seen anything yet.
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30
14-:-XX:G1HeapRegionSize=16M
14-:-XX:+UseStringDeduplication
## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err-pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
Any other G1GC tuning we could do?
Thanks