Thank you for the link.
In my opinion, the best way to do java performance monitoring is to instrument bytecode at loadtime [ by using an agent packaged as a JSR 163 compliant "javaagent" and added via jvm option ] which allows to weave aspects into application code without having to modify its source code, or even to control the build process.
There are many techniques to achieve that, one of them is to use the aspectJ weaver.
I'm working on this java-agent open-source project (https://github.com/leadwire-apm/leadwire-javaagent) .
It's a fork from the kieker project which is a good and performant framework for java performance monitoring.
Now I'm thinking to create a new java writer that sends data directly to apm-server over the Transaction API.
There is some points to consider :
- performance : generating complex and heavy payloads (composed of thousands of method calls and sql statements ...) may induce too much overhead on the jvm and increases memory retention [ we have to wait to gather all the parts of a transaction before sending payload ] ; even calculating durations takes some overhead ...
- scope : In case of distributed transactions on multiple jvms, every agent sees only a part of the global transaction; so we need to have a pre-processor that assembles the parts
I think that the approach of going through a pre-processor (calculate durations, gather transaction parts) is preferable in case of very heavy loads or complex distributed applications.