Java APM agent integration is not showing external request/transaction data

Elastic Observability APM
,
1

My Elasticsearch stack version is 8.14.0 and APM agent version installed on edge machines is 1.51.0.

I'm able to capture the data from the application and visualize in Kibana under custom dashboard created. However, It is displaying the data related to internal requests/transactions only not the external/outgoing transactions. There is transactions captured related to external/outgoing requests under APM in kibana. Below is my APM agent configuration on edge machines-

java -javaagent:/path/to/elastic-apm-agent-1.51.0.jar \
-Delastic.apm.service_name=my-service-name \
-Delastic.apm.secret_token= \
-Delastic.apm.server_url=http://x.x.x.x:8200 \
-Delastic.apm.environment=my-environment \
-Delastic.apm.application_packages=org.example,org.example* \
-jar my-service-name.jar

How can I enable monitoring the above mentioned requirement from the APM agent?

Appreciate you help in advance. Thanks!

2

What do you mean by "internal" and "external" transactions ?

In the APM data model, we use the following model

  • transactions are the main "work unit" observed, for example when a call is made to an HTTP endpoint, or when a message is received from a messaging queue.
  • spans are either internal "work units" to a transaction or outgoing calls to downstream services (call another HTTP server with an HTTP client, call to a database, ...)
  • a trace is an aggregation of multiple transactions and spans across multiple services to have an end-to-end tracing of a distributed task in the observed system.

So from you message I guess that you are able to monitor the HTTP endpoint, and you get one transaction captured for each incoming HTTP call to your service.

What could be missing is the calls to other internal services, and for that you need to check that the technologies used to communicate with them are part of the supported technologies.

Also, as an alternative you could also try using the OpenTelemetry Java agent or the Elastic distribution as it might support out of the box the libraries that you are using. Please note that the configuration will need to be adjusted.
See documentation for more details on this.

3

Hello,

Internal transactions are nothing but the api calls(http) made to another microservice which is in the same network which is also instrumented.

External transactions are nothing but the api calls(https) made to remote endpoints of another microservice residing in different network.

Thanks

4

The calls to HTTP or HTTPs endpoints should be instrumented and captured as spans. They also should be part of a transaction to be captured.

There is no distinction between internal or external endpoints on the network/url, the only difference is the HTTP client that is used to make the calls. If both are using the same HTTP client, for example Apache HTTP Client, then they should all be captured properly.

5

Earlier, The same instrumentation worked with the Java APM agent version 1.31.0 and Stack version 8.6.2. When it is upgraded to APM agent 1.51.0 and Stack version 8.14.0, half of the dashboard data are unavailable. Kept the same and minimal config for APM agent.

Please suggest me how I can debug this.

6

We need to know where the issue is, so I would suggest to use the 1.31.0 agent version with your current stack and see how it behaves.

Then, if the issue persists it means it's not related to the agent, but more to the data ingestion or the stack deployment.

If the issue is gone, then it's likely to be related to the agent. In this case it would be relevant to capture agent debug logs using this procedure and do it with the two agent versions and compare the logs to spot any difference.

7

One important thing here, We had an the stack Java APM agent version 1.31.0 and Stack version 8.6.2 which completely stopped working for some reason(data in the APM and custom dashboards are empty). We were unable to find the root cause for it.

So, Later we have upgraded it to APM agent 1.51.0 and Stack version 8.14.0 as part of fixing it.

8

Sorry, but the symptoms are a bit vague and it's very hard to understand what could be wrong here.

How did you investigated the first time it failed, what steps of the troubleshooting guide did you follow ? Were there any log messages in the agent log that could indicate the cause for this missing data ?
Also, on the stack side, was there any hint in the logs of what could happen ?

So maybe it's better to start investigating with your current setup, with the 1.51.0 agent version and the stack version that you have.
With that, start by investigating the agent logs, then if there is no visible error/warning then start investigating on the stack side.