ES 6.8.23 too many Full GC

[Andrey_Tyutyunov]

Good day!

Version 6.8.23
3 nodes, 26GB heap per node, 64GB RAM on each node
java 14
Indices: 49
Documents 516,257,116
Disk Usage 872.0 GB
Primary Shards 165
Replica Shards 165
Search Rate (/s) ~ 750 /s (total shards)
Indexing Rate (/s) ~ 200 /s (total shards)

On garbage collector stat I see strange situation - too many cases Full GC, below Full GC on 2 hrs 24 min 10 sec

Full GC stats

Full GC Count	12328
Full GC reclaimed	7.81 gb
Full GC total time	40 sec 782 ms
Full GC avg time	3.31 ms
Full GC avg time std dev	2.60 ms
Full GC min/max time	0.466 ms / 15.6 ms
Full GC Interval avg	701 ms

It's okey? Looks very strange. I have no see any other problem, elasticsearch log without errors / warnings / causes, response time - search time - index time no any problems, CPU usage on any node 20-30%, each node on deducated server.

[Andrey_Tyutyunov]

I think I found answer - this is look like bug in GC log visualizer for second rotated log, first gc log looks similar but all marks as Young GC, correct me please if I'm wrong

[warkolm (Mark Walkom)]

Your GC looks pretty good, it's running and cleaning heap as it should be.

@Andrey_Tyutyunov

Version 6.8.23

Please upgrade 6.8 is very much past EOL and there have been numerous improvements in newer versions that will help your situationm

[Andrey_Tyutyunov]

Thank you, upgrade in progress - the cluster uses in heavy load project as main store and need in refactor code and reorganize DB