Kibana APM not showing @CaptureSpan and @CaptureTransaction

Elastic Observability APM
21

Hi again,

Since I feel it is necessary in any case, I did a PR to log the location of the classes that agent causes loading and with the ability to disable the checks we do that may raise this problem. You can download this snapshot jar and test it.

With this snapshot, if you set the agent log level to DEBUG it should print out where it found the javax/ws/rs/Path class.
In addition, by adding -Delastic.apm.enable_class_loading_pre_filtering=false to the agent config you would disable the code that I suspect is related to it and see if this solves the problem. The effect would be longer startup time, but not necessarily very considerable, depending on your app.

22

Hi Eyal,

I downloaded elastic-apm-agent-1.5.1-SNAPSHOT.jar

My JAVA_OPTS in catalina.sh are now:

JAVA_OPTS="-DTOMCAT1 -verbosegc -Xms64M -Xmx8G -XX:MaxPermSize=1024M -noverify -DQES_DIR=$SOFTWARE/conf -Dfile.encoding=UTF-8 -Dlog4j.configuration=file://$SOFTWARE/conf/log4j.properties -Dorg.owasp.esapi.resources=$SOFTWARE/conf -javaagent:/usr/software/apm-client/elastic-apm-agent-1.5.1-SNAPSHOT.jar -Delastic.apm.service_name=qeswebmonitor -Delastic.apm.server_url=http://localhost:8200 -Delastic.apm.application_packages=com.qwam -Delastic.apm.log_level=DEBUG -Delastic.apm.enable_class_loading_pre_filtering=false "

If I remove jersey-core-1.9.jar from mypapp/webapp/WEB-INF/lib my app refuses to deploy with a NoClassDefFound with or without your new filter:

Logs: Without jersey without filter

Logs without jersey, with filter

With jersey-core-1.9.jar and your filter disabled , the app deploys ...

Logs: With jersey and your filter disabled

... we are back to the former situation we had but with no more explanations about which jar it is conflicting with. Besides, I do not care much about that, since it works now.

Still no trace of my Spans and Transactions annotations in Kibana.

23

Then maybe there are no conflicting jars, and this comes from something in the way WebappClassLoaderBase is looking for already loaded classes that prevents it from finding a defined class even if it uses the same jar. In any case, this last configuration only means a bit longer startup time, but same functionality is expected so it should be good enough.

Still no trace of my Spans and Transactions annotations in Kibana.

Right, that's what we wanted to reproduce in a clean environment, and this is done. What I saw in prior logs was that the xmlToFacets appeared in the stack trace of a traced transaction. Would you please run again the same configuration only with TRACE logging level instead of DEBUG so that we can the log with stack traces on the clean env?

24

Hi Eyal,

I have a difficult thing to say:

I did not notice but the source file I put the annotation in, is in groovy :hot_face:
I did not notice the extension of the file, and nothing looks more like a Java source file than a groovy source file. This projects uses more and more Groovy since our developers favours it.

I guess that groovy is not supported yet.

I apologize for this Eyal since I should have told you this first thing , but I missed it.

This explains that everything works due to the java compatible code generated but a tiny thing must me missing somewhere in the instrumentation, due to the groovy compilation, preventing the annotations to work properly.

I do not know if you still want to talk to me. Here are the logs with TRACE enabled as you requested it

25

Ivan,

No worries, I am glad we have a reasonable explanation, and you found an issue I fixed anyway, which is great :smiley: .

Do you compile the groovy to Java bytecode? If so, maybe all you need is make sure that the API jar is included in the compile classpath, like explained in this groovyc documentation?

Regards,
Eyal.

26

Sorry Eyal, but I do not get what you mean.
On my desktop, where we are running thoses tests, I am using the IDE Intellij from Idea with the basic groovyc compiler. This compiler generates stubs, which are the source classes translated from groovy to Java and then compiles it with javac (as far as I understand it).
There is not much that I can do to interfere in the process.

27

Ivan,

I believe there is a place to configure the groovyc compile classpath in the IDE, but I won't be able to assist with locating that.
If you find a way for the annotations to make their way to the compiled Java bytecode, you should be able to get your methods traced.

Let me know if there's anything else I can do.

Regards,
Eyal.

28

Generally annotations make their way through, since we have other Spring annotations which make their way through, both in Groovy and Java.

I took enough of your time and thank you again for your kind and extensive support Eyal.
I will wait for the moment when APM fully support groovy, APM is already helping us much as is.

Please close the case.

29

This is not a problem at all.
I assume this is only a matter of definition- either in the project dependencies or for the groovyc, but I will leave this to you :slight_smile:.
I am very glad that you find our solution useful!
All the best,
Eyal.

30

I found a few weeks ago that the AOP filters for ByteBuddy are ignoring everything that comes from a groovy classloader, so it's not a matter of simply making sure the api jar is in the classpath.
Source: https://github.com/elastic/apm-agent-java/blob/aaccd273c241a4b22da6d4bdb3e09cf5ed8bf37b/apm-agent-core/src/main/java/co/elastic/apm/agent/bci/ElasticApmAgent.java#L308

I've had to create my own aspect to mimic @CaptureSpan and @CaptureTransaction for our projects, and it's limited because it only works on spring beans. To make it global, I'd have to link another javaagent via VMOptions, which is bad considering I already have the apm java agent on.

I'd love to see the apm-agent-java gain official support for groovy, even if it's only by making sure it tries to work the annotations in a groovy classloader. I'd be willing to test a snapshot, considering I already have some projects up for it on the process of going through full observability.

31

@fredgalvao Thanks for the feedback!

Are you referring to groovy precompiled to bytecode, or interpreted at runtime?
If the latter- yeah, I assume this would require specific support.
If the former- can you provide a compiled class, or check yourself if the annotations get there at all?

In either case, did you consider using our explicit APIs, instead of annotations? You can achieve the same things (and more) with those.

Lastly, if you DO eventually develop something useful and want to share with the community- talk to us. We will love to assist with contributions. Currently, formal support for groovy is not on the immediate roadmap not because we don't think it is important, but because of other stuff prioritized higher.

Cheers,
Eyal.

32

Hi Eyal,

In our configuration we are in fully compiled groovy with @TypeCheck annotation enabled
Following your suggestion to use the explicit API, I shifted from the annotation to the explicit API by adding this at the beginning ....

Transaction transaction = ElasticApm.startTransaction();
transaction.setName("ResultatService#retrieveAll()");
transaction.setType(Transaction.TYPE_REQUEST);

... and this at the end of our method

transaction.end()

.. but "ResultatService#retrieveAll()" do not show up in Kibana.

33

If ResultatService#retrieveAll() is already part of an existing transaction, use ElasticApm.currentSpan().startSpan(), instead of creating another transaction.

Otherwise, could you upload the full debug logs of the agent, including the whole startup, up until the first invocation where ResultatService#retrieveAll() is supposed to be part of? Also wait until the reporter is flushed (10s) so that the logs contain the APM Server response.

34

I apologize for the API misuse.
I shifted my code to ...

Span span = ElasticApm.currentSpan().startSpan()
span.setName("ResultatService#retrieveAll()")

span.end()

... and It works !!!!!! :star_struck:

I can now clearly see my spans in Kibana which leverages the great power of APM giving us insights of our app performance that we never had before.

So for compiled Groovy with @TypeCheck enabled, I validate that the explicit API works !

Thank you Felix and Eyal for your great help

3 Likes
35

To begin with, what I had initially done was create my own marker annotations with partial support for AOP (that would only work on spring beans, and it wasn't even 100% of all cases), so that I could use it on our groovy codebase (edit: to use the explicit API around code with ease).

Now, I may have very good news. Please see, vote, comment, test, and support this: https://github.com/elastic/apm-agent-java/pull/567

1 Like
36

@ivanqwam, Now that the PR was merged, you should test it with a snapshot version and see if it works for you on that groovy code, so that you don't have to use the explicit API.

Also, @Eyal_Koren is there any plan to do a release with that soon?

37

Yes, we’ll release soon. Make sure to subscribe for releases on GitHub.

38

All,

I downloaded elastic-apm-agent-1.5.1-20190415.093919-19.jar

I replaced explicit calls by @CaptureSpan

and .... it works now !

Spans are now showing in Kibana APM dashboard

We are using groovy-2.4.14

Congrats Frederico !

2 Likes
39

@ivanqwam @fredgalvao Thanks a lot for your efforts and wonderful contribution! :elasticheart:
Version 1.6.0 will be out today or tomorrow and should contain the fix "officially" :tada:

1 Like
40

You should mark the topic as fixed, I guess. Maybe even change the title to better represent that it was an incompatibility with groovy generated code, to help future developers find it easier.

Also, I'm happy I could help!