APM Java agent not creating transactions with Vert.x even with experimental features enabled

Elastic Observability APM
1

Hi!

I am not being able to audit the transactions against my Vert.x application using the Java agent for this purpose, even though I have enabled the experimental instrumentations.

-Dquarkus.profile=local
-javaagent:src/main/resources/elastic-apm-agent-1.36.0.jar
-Delastic.apm.secret_token=TOKEN
-Delastic.apm.service_name=get-started
-Delastic.apm.server_url=SERVER_ENDPOINT
-Delastic.apm.enable_experimental_instrumentations=true

It's auditing the app metrics but no transactions.
The application logs also don't mention any transaction being created.

The versions are:

APM Java Agent: 1.36.0
Vert.x Web: 4.3.7
Java: 
- Java 11.0.17 Runtime version: 11.0.17+8 VM version: 11.0.17+8 (Eclipse Adoptium) 
- Java 17.0.6 Runtime version: 17.0.6+9-LTS-190 VM version: 17.0.6+9-LTS-190 (Oracle Corporation)

Do you have any idea what the problem could be?

2

Hi @missael.denadai welcome to the community!

Try enabling DEBUG or TRACE and looking closely at the agent logs (which will be part of the app logs)

Is the agent successfully connecting to the APM SERVER?
Are there JVM Metrics showing up in Kibana?

-Delastic.apm.log_level=DEBUG

3

Hi @stephenb thank you for replying.

The agent successfully sends JVM metrics to the APM Server.
I enabled TRACE as you suggested and nothing is logged when I send HTTP requests for the application.

I noticed, although, errors when the agent was trying to create some specific threads. The stack trace is always as following, except the name of the thread:

2023-03-16 15:47:59,378 [main] DEBUG co.elastic.apm.agent.util.ExecutorUtils - A new thread named `elastic-apm-configuration-reloader` was created. The original context class loader of this thread (jdk.internal.loader.ClassLoaders$AppClassLoader@251a69d7) has been overridden
2023-03-16 15:47:59,382 [main] TRACE co.elastic.apm.agent.util.ExecutorUtils - Stack trace related to thread creation: 
java.lang.Throwable: null
	at co.elastic.apm.agent.util.ExecutorUtils.logThreadCreation(ExecutorUtils.java:141) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at co.elastic.apm.agent.util.ExecutorUtils$SingleNamedThreadFactory.newThread(ExecutorUtils.java:130) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:630) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:920) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1593) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:632) [?:?]
	at co.elastic.apm.agent.impl.ElasticApmTracerBuilder.scheduleReloadAtRate(ElasticApmTracerBuilder.java:197) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at co.elastic.apm.agent.impl.ElasticApmTracerBuilder.build(ElasticApmTracerBuilder.java:153) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at co.elastic.apm.agent.impl.ElasticApmTracerBuilder.build(ElasticApmTracerBuilder.java:133) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:162) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:161) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:102) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at co.elastic.apm.agent.premain.AgentMain.premain(AgentMain.java:51) [elastic-apm-agent-1.36.0.jar:1.36.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491) [?:?]
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503) [?:?]

I also tried to downgrade the Vert.x version to 3.9.15 by downgrading the Quarkus version. No transactions either.

I also tried using the public API by adding the apm-agent-api:1.39.0 dependency, but the annotations didn't work and also the result of any ElasticApm.startSomething() was a noop entity.

Any leads?

4

Well, it is experimental :slight_smile:

Not immediate ... and many of the Engineers are in a Conference I will try to ping one but I suspect he will not be able to look until next week.

@Eyal_Koren Any Thoughts?

5

TRACE is too detailed, use DEBUG. If you are using annotations, you need to set the application_packages or the annotations are ignored (per the documentation)

1 Like
6

The vert.x instrumentation Captures incoming HTTP requests as transactions do you have any of those incoming requests?

1 Like
7

Hi @Jack_Shirazi , thank you for your reply.

Yes, I send HTTP requests to my application. The application responds the requests, but the transactions are not being created by the agent(no logging about the agent starting transactions).

Any chance you guys have an example of a working Java agent on a Vert.x Web application?

8

Looks like you are using Quarkus. Do you use it in a native image mode?

9

Hi @Eyal_Koren
No, we are not using the native compilation.

10

Can you see the following line in your DEBUG log:

Method match for instrumentation VertxTracerReceiveRequestInstrumentation

?

If VertxServerTest is not what you are looking for, look for other tests in the area.

There is also a PR with Quarkus tests, which is not up-to-date but shows that we generally support it, with a small modification required.

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.