Java agent

Elastic Observability APM
#1

Hi,

I implemented a java agent which works very fine with the ELK Stack.
The process is very simple :

  • the agent writes performance traces on filesystem or over network
  • a javascript agent (injected by the java agent) sends back real-user-monitoring data (a http request)

=> All this data is handled by logstash and integrated into Elasticsearch.

I'm very exited about the elastic-apm and trying to use it instead of logstash, but i need to know some specifics of the Transaction API :

  • Will the apm-server accept new agent types (other than elastic-node, python ...) ?
  • Can I send separately different parts of a transaction (the javascript rum part and the java part) or do I have to wait to get all the parts ?
  • How to generate a payload for a distributed transactions (wich runs on multiple jvms/multiple hosts) ?
#2

I have implemented a POC for the JVM. As of now it works for the Spring framework.
It keeps data on memory queues and send asynchronously to the apm server.
The source is open: you can reach it at https://github.com/davidecavestro/elatic-apm-java-agent-poc5
Good luck

#3

Thank you for the link.

In my opinion, the best way to do java performance monitoring is to instrument bytecode at loadtime [ by using an agent packaged as a JSR 163 compliant "javaagent" and added via jvm option ] which allows to weave aspects into application code without having to modify its source code, or even to control the build process.

There are many techniques to achieve that, one of them is to use the aspectJ weaver.

I'm working on this java-agent open-source project (https://github.com/leadwire-apm/leadwire-javaagent) .
It's a fork from the kieker project which is a good and performant framework for java performance monitoring.

I took form the kieker project the part that works with the "JSR 163" paradigm, and added some extra futures like the injection of javascript on servlet calls (for real user monitoring) , the system resources monitoring and sql traces gathering.

Now I'm thinking to create a new java writer that sends data directly to apm-server over the Transaction API.
There is some points to consider :

  • performance : generating complex and heavy payloads (composed of thousands of method calls and sql statements ...) may induce too much overhead on the jvm and increases memory retention [ we have to wait to gather all the parts of a transaction before sending payload ] ; even calculating durations takes some overhead ...
  • scope : In case of distributed transactions on multiple jvms, every agent sees only a part of the global transaction; so we need to have a pre-processor that assembles the parts

I think that the approach of going through a pre-processor (calculate durations, gather transaction parts) is preferable in case of very heavy loads or complex distributed applications.

#4

Really interesting!
The AOP approach, and specifically bytecode instrumentation through jvm agent is a fantastic way to go.

For sure the Java ecosystem is plenty of resources coming from the huge investments of the past decades. i.e. during last year I've used Jolokia, which proxies jmx beans over http both with and without jvm agents.

Then I guess it depends on the use cases you want to support, the kind of data you want to gather and the probes you are allowed to use.

i.e. I expect some people don't want to cope with jvm agents at all: maybe they have an on-premise JEE application server and prefer to configure AOP on a per-classloader fashion.
Or maybe they have a completely automated delivery system to deploy their microservices on lightweight containers on the cloud, but still prefer to completely avoid proxies and instrumentation and would decide to simply write some custom traps.
IMHO an interesting approach could be providing support for plugging monitoring libraries, as happens for http/rest client libs.

About transactions distributed on multiple jvm instances/hosts I'd say you need a separate stage to aggregate (through existing correlation ids?) the data you previously gathered and saved (into ES?).
I guess it would be better having that stage outside the monitored application (into logstash?).
But maybe I'm describing what you already have.
And I hope some guy from the apm team will give you some insight... more useful than my speculations :slight_smile:

#5

I forgot to mention JFR. So far I used it "only" from Mission control: very effective, low overhead.
It could be worth checking http://hirt.se/blog/?p=870

Please note it usually needs to unlock commercial features on the oracle jvm (that in turn implies paying some license feed).

#6

Hi Wassim and Davide,

Thank you both for your interest in Elastic APM!
We appreciate your efforts to build agents - and we know it's a simple easy task :slight_smile:

@wassim.dhib to answer some of your questions:

@davidecavestro great job on that POC for the JVM!

For 6.2, be sure to check the breaking changes list here: https://github.com/elastic/apm-server/blob/master/CHANGELOG.asciidoc

I also wanted to mention that we are going to be working on an official Java agent. It will be open source and we'd love to have your input once it's out.

2 Likes
#7

Sure!
Please let me know if I can contribute also with some coding.

1 Like
#8

Hi,
I am also interested to participate in the development of an official Java agent, I have experience in java performance tuning, troubleshooting and testing.

3 Likes
#9

Hi, i am very eager with this good news.
Please, keep us posted.

#10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.