Logstash degraded when jvm heap size is near its limit

That's hard to explain. With -XX:CMSInitiatingOccupancyFraction=75 and heuristics disabled using UseCMSInitiatingOccupancyOnly CMS sweeps should be kicking in as soon as heap usage crosses 3 GB. Normally that would, in fact, limit heap usage to around 3 GB, meaning you never get to use 25% of the heap. That's why Sun gradually increased the default occupancy trigger threshold from the original ridiculous default (something like 66%) up to, I think, 96%.

Your heap usage is very close to reaching 100% before it drops down, which suggests to me CMS is not freeing stuff, and a Full GC is triggered. At the full GC the JVM may have freed up weak references that are used for cached items, which could conceivably impact performance, but I would not expect that impact to persist. Anyways, to improve CMS's performance it is possible you would need CMSPermGenSweepingEnabled and CMSClassUnloadingEnabled, but we need GC logs to confirm it.

Can you add

-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintGCApplicationStoppedTime

to the jvm.options, restart logstash and run with that logging enabled until the performance degrades, then post the log somewhere so that we can look at it?