I have implemented a POC for the JVM. As of now it works for the Spring framework.
It keeps data on memory queues and send asynchronously to the apm server.
The source is open: you can reach it at https://github.com/davidecavestro/elatic-apm-java-agent-poc5
Good luck
In my opinion, the best way to do java performance monitoring is to instrument bytecode at loadtime [ by using an agent packaged as a JSR 163 compliant "javaagent" and added via jvm option ] which allows to weave aspects into application code without having to modify its source code, or even to control the build process.
There are many techniques to achieve that, one of them is to use the aspectJ weaver.
I'm working on this java-agent open-source project (https://github.com/leadwire-apm/leadwire-javaagent) .
It's a fork from the kieker project which is a good and performant framework for java performance monitoring.
I took form the kieker project the part that works with the "JSR 163" paradigm, and added some extra futures like the injection of javascript on servlet calls (for real user monitoring) , the system resources monitoring and sql traces gathering.
Now I'm thinking to create a new java writer that sends data directly to apm-server over the Transaction API.
There is some points to consider :
performance : generating complex and heavy payloads (composed of thousands of method calls and sql statements ...) may induce too much overhead on the jvm and increases memory retention [ we have to wait to gather all the parts of a transaction before sending payload ] ; even calculating durations takes some overhead ...
scope : In case of distributed transactions on multiple jvms, every agent sees only a part of the global transaction; so we need to have a pre-processor that assembles the parts
I think that the approach of going through a pre-processor (calculate durations, gather transaction parts) is preferable in case of very heavy loads or complex distributed applications.
Really interesting!
The AOP approach, and specifically bytecode instrumentation through jvm agent is a fantastic way to go.
For sure the Java ecosystem is plenty of resources coming from the huge investments of the past decades. i.e. during last year I've used Jolokia, which proxies jmx beans over http both with and without jvm agents.
Then I guess it depends on the use cases you want to support, the kind of data you want to gather and the probes you are allowed to use.
i.e. I expect some people don't want to cope with jvm agents at all: maybe they have an on-premise JEE application server and prefer to configure AOP on a per-classloader fashion.
Or maybe they have a completely automated delivery system to deploy their microservices on lightweight containers on the cloud, but still prefer to completely avoid proxies and instrumentation and would decide to simply write some custom traps.
IMHO an interesting approach could be providing support for plugging monitoring libraries, as happens for http/rest client libs.
About transactions distributed on multiple jvm instances/hosts I'd say you need a separate stage to aggregate (through existing correlation ids?) the data you previously gathered and saved (into ES?).
I guess it would be better having that stage outside the monitored application (into logstash?).
But maybe I'm describing what you already have.
And I hope some guy from the apm team will give you some insight... more useful than my speculations
I forgot to mention JFR. So far I used it "only" from Mission control: very effective, low overhead.
It could be worth checking http://hirt.se/blog/?p=870
Please note it usually needs to unlock commercial features on the oracle jvm (that in turn implies paying some license feed).
I also wanted to mention that we are going to be working on an official Java agent. It will be open source and we'd love to have your input once it's out.
Hi,
I am also interested to participate in the development of an official Java agent, I have experience in java performance tuning, troubleshooting and testing.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.