Instrument custom methood and create a transaction without code changes

Elastic Observability APM
,
1

APM version 8.10.3

we are using Vault secrets management solution for our java applications. We have issue with vault server overloaded and returning http error. Our java application usess Vault java driver which uses java.net.URLConnection to make call to vault server. Added apm java agent to our application and see only errors from vault. Not able to get any trace with vault method even though i have added custom trace for vault driver class and method. If vault http call is running in background thread, how do we get complete call stack with number of calls, dependency overview for vault and errors to get complete context. We are trying to get how many calls made from our application to vault using vault driver along with dependency stats for vault.
Can we add a method via Trace_methods and start transaction instead of span as its running in background thread and not being captured under any of the transaction. Thanks

2

Hi,

With Elastic APM, the root of a distributed trace is a Transaction, which is usually the entry point of the application (incoming HTTP method call for an HTTP server for example).

In your case, the calls to Vault are likely executed outside or any transaction, for example when the application starts. In order to fix that you'll have to use trace_methods to instrument a method that wraps the HTTP calls to Vault.

For example, if your application has a single method that includes the whole application initialization where the calls to Vault are made, then it would be a good candidate for instrumentation with trace_methods.

trace_methods will create both transactions or spans depending on the execution context: if there is no active transaction a transaction is created, if there is one a span will be created.

As an alternative, with OpenTelemetry do not make any distinction between the spans and transactions, so in your case it would capture the outgoing spans without requiring to have a "parent span" that wraps the HTTP calls to Vault.
However, this would require to use the OpenTelemetry java agent instead of the Elastic APM agent.

3

Thanks for the detailed info. I did try adding method with trace_methods config property. But i dont see new transaction created with custom trace method added for Vault

4

In order to know if the trace_methods is applied properly you should do the following:

  • when the agent starts, the effective configuration is shown, you should see the value you have set for trace_methods configuration. If that's not the case, then it means it's not configured properly.
  • set the agent with log_level=debug, with that the agent will log the instrumented methods if you grep for lines that include Method match for instrumentation you have all the instrumented methods. The method you have configured should be part of them, if that's not the case then it means the value that is configured does not match the application code.
  • if the method is properly instrumented, then you should see startTransation and endTransaction in the log messages when the method stars and ends.
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.