Hi, @danielmitterdorfer,
Sorry for the late reply, it took me some time to make the tests.
- Switching to CMS solved the issue. The heap size never got big and the parent circuit breaker was never triggered.
Back to G1GC:
I ran the scenario again, waited for the parent circuit breaker to be triggered and watched the gc.log file. It took it minutes to come back to life. During that time there were not GCs in the gc.log file! It seems that the GC already freed the memory and somehow the process thinks that it's still taken.
I got the feeling you are doing your tests against JDK11 (see this discussion). I replaced the bundled JDK with the 11 version. Things were much better. Even though that the memory was cleaned after 75% it was still cleaned when it ran out of memory and the circuit breaker wasn't triggered. But even with it after several tries it was triggered but only for a short period of time and then the memory was cleaned and the process could process commands.
So my questions are:
- Are you using CMS in your internal tests?
- Is JDK 12 also used during tests or only JDK 11?
- A bit of a different topic, is there a reason why the stats were also affected by the circuit breaker? It makes debugging the issue harder.
Eli