APM Java agent not capturing background running details and methods when enable sampling profiler

[Bhanu_Praveen]

Hello,

I have enabled sampling profiler, only UI transactions and its stack trace methods are being captured. But if i have any background schedulers/processes running, those events are not being captured.

v 7.8.0:

v 7.8.0:

apm-server-7.12.0:

Java elastic-apm-agent-1.24.0:

Chrome:

Installed from source:

Fresh install

Is there anything special in your setup?nope

I have enabled sampling profiler, only UI transactions and its stack trace methods are being captured. But if i have any background schedulers/processes running, those events are not being captured.:

Kindly let me know how to capture all background running async processes details and methods?

Thanks

[Eyal_Koren]

If you are using a supported scheduling framework, then maybe setting the application_packages is all you need to do.

Otherwise, you would need to manually trace those. One option is using the trace_methods config option, but make sure to be very specific in what you set through it, don't cast a too-wide net. Another option is to manually trace the relevant methods through code, by using our public API.

[Bhanu_Praveen]

Thanks for your reply, not using any framework. I am new to this ELK APM agent. Trying our for first time. When i enable profiling_inferred_spans_enabled, then all methods should be captured by default? Right? If i want to deploy this in prod and need to capture all CPU intensive methods, i can't give any specific. Which ever methods are highly occupied, all those should be captured for me.

Kindly suggest

Thanks

[Eyal_Koren]

When i enable profiling_inferred_spans_enabled , then all methods should be captured by default? Right?

Just to be clear - all methods can potentially be captured by default. We definitely don't want to capture all executed methods for performance and usability reasons. Instead, the agent will look at frequently sampled stack traces and create spans only for methods identified as hot spots.

However, for this to happen, there must be an active transaction on the thread, which is why I proposed ways of making sure of that. It's enough to start and activate a transaction at the beginning of the task and end and deactivate it at the end of the task for the sampling profiler to do what it does.

I hope this helps.

[Bhanu_Praveen]

Hello,

I have a run a test with async xml's processing which will happen through backend scheduler services. Enabled jmx and able to get all below methods with the Jmc as seen in below picture.

Now i have taken some packages out of Jmc which were already captured and added below arguments and ran the same test again.

-javaagent:/path/to/elastic-apm-agent-1.24.jar
-Delastic.apm.service_name=myApp -Delastic.apm.application_packages=com.mdi.trade.gpm,com.mdi.rulesengine,com.mdi.core.db
-Delastic.apm.server_urls=http://xx.xx.xx.xx:8200

Still not able to capture any methods through the APM. Only UI http calls are being captured.Our async service will create threads and get processed, can we get Thread stacktrace?

Kindly let me know any other configurations need to be done?

Thanks

[Eyal_Koren]

Are you using a scheduling framework within your Java service, or a different service that then invokes requests to your Java service?
Please go over our supported frameworks page and let us know ones you are using in your app.

[Bhanu_Praveen]

Hello,

We are using log4j, Springboot, Servlets etc in our code base. Asynchronous threads will process the Xml's which will go through some rules(combined methods flows) and then complete processing. Can we capture any thread stacktrace? Bcoz in Visual VM, when we capture CPU sampling, it goes through consumer Thread stacktrace.

Thanks

[Eyal_Koren]

See my comment above.

Since you use a non-supported scheduling framework, you must create transactions so that anything will be traced. Our profiler ignores stack traces from threads that don't have active transactions.

You can use our trace_methods config to specify method name/s to use as the base for transaction creation.
If the XML processing code is yours, you can also use our public API within your code on the proper methods to capture transactions.

[Bhanu_Praveen]

I have added trace_methods

-Delastic.apm.trace_methods=processGPM,execute,executeRule,processWork

Still not able to capture any methods. We don't want to update code for instrumentation.
Kindly let me know any other ways we can enable?

Thanks

[Eyal_Koren]

Review carefully the trace_methods documentation and the given examples.
You need to provide fully qualified name of the classes also, not only the method names.
You can use wildcards, but I strongly suggest you don't, be as specific as possible with the methods you trace.
Good luck, I hope this works out!

[Bhanu_Praveen]

Tried with full Method path, still no luck. Anyway we can get stack trace for running Threads? As our consumer threads will process Xmls. In Visual VM, we go through the Thread stacktrace.

Thanks

[Eyal_Koren]

@Bhanu_Praveen

Anyway we can get stack trace for running Threads?

We capture stack traces only on spans that cross a threshold and using the sampling profiler only for threads that have active transactions on them. We don't randomly capture and send stack traces of all threads.

Please share the followings, so we can assist:

1. You entire agent setup and configuration
2. The code of a method you try to trace
3. Set log_level to debug and share the log from startup and until the first schedules task is invoked. You can share this through https://gist.github.com

[Bhanu_Praveen]

Hello,

Its not a single code file, that i can share. Its a framework which will traverse through multiple packages. Kindly let me know how can i share the whole folder?

Thanks

[Eyal_Koren]

Let's start with one - pick a method you want to trace and share the related code, as well as the other requested input (full agent setup and config, debug-level log).