Logstash degraded when jvm heap size is near its limit


I have a logstash performance degradation problem. It happens when jvm heap memory size is near its upper level. Events received/emitted rate drops sharply and latency increases dramatically. Only after service restart - all indicators returns to normal.

Here is my jvm.options setting:





Kibana minitoring pic:

That's hard to explain. With -XX:CMSInitiatingOccupancyFraction=75 and heuristics disabled using UseCMSInitiatingOccupancyOnly CMS sweeps should be kicking in as soon as heap usage crosses 3 GB. Normally that would, in fact, limit heap usage to around 3 GB, meaning you never get to use 25% of the heap. That's why Sun gradually increased the default occupancy trigger threshold from the original ridiculous default (something like 66%) up to, I think, 96%.

Your heap usage is very close to reaching 100% before it drops down, which suggests to me CMS is not freeing stuff, and a Full GC is triggered. At the full GC the JVM may have freed up weak references that are used for cached items, which could conceivably impact performance, but I would not expect that impact to persist. Anyways, to improve CMS's performance it is possible you would need CMSPermGenSweepingEnabled and CMSClassUnloadingEnabled, but we need GC logs to confirm it.

Can you add


to the jvm.options, restart logstash and run with that logging enabled until the performance degrades, then post the log somewhere so that we can look at it?

Thank you so much for your detailed response!

I turned on GC logging as you requested. When the problem appears, I'll share the log..


It happened. But something more strange this time (see pic.).


Monitoring pic:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.