[ES v2.3] jvm keep gc with no index and query, slow down whole cluster

Yu_Han · January 8, 2018, 3:06am

we have a huge cluster. 5 nodes.
about 15000000000 documents
and 15000 shards.

jvm:
ES_JAVA_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=10M -XX:+CMSIncrementalMode -XX:+CMSScavengeBeforeRemark -XX:+ParallelRefProcEnabled -XX:NewSize=5G -XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=20 -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly"

it keeps gc, looks like figure below

after delete half data, reduce both shards count and ducuments size. it becomes health.
I am querying what is the root cause? is there any limit for shards count or index size on a node.

what metadata maintained in memory, cause occupy huge memory when cluster goes huge.

dadoonet · January 8, 2018, 3:23am

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Too many shards per node seems to me one of the good reasons. Then it can be caused by fielddata as well. Depends on your mapping and if you are using doc values or not.

Also, you should upgrade.

But the first thing to do is to reduce the number of shards per node.

Have a look at

system · February 5, 2018, 3:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.