RestTemplate Calls inside another RestTemplate's ClientHttpRequestInterceptor is not seen in as a span

Elastic Observability APM
,
1

I have a Spring Boot application which has an endpoint /serviceCall that makes REST call to an external endpoint (for example : http localhost:8200/config/v1/agents?service.name=demo-service) via RestTemplateA. This RestTemplateA has an interceptor which is used to get token via a Spring Component TokenService. TokenService has another RestTemplate that makes a request to another external endpoint (for example: http localhost:8200) to get a token and set it as header token to RestTemplateA call. This transaction is recorded by Elastic APM Java agent, however the first call to get token (http localhost:8200) is not in the span.

When I make token call directly via /directCall endpoint, same request is recorded as expected.

Is there a way to get the rest calls recorded when they are made from an ClientHttpRequestInterceptor instance? I shared a demo project on github

Kibana version: 8.16.1

Elasticsearch version: 8.16.1

APM Server version: 8.16.1

APM Agent language and version: Java 1.52.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version: From Docker repository

Fresh install or upgraded from other version? Fresh Install on my local docker desktop environment

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. Install and setup Elasticsearch, Kibana and APM Server as stated in APM Server binary | Elastic Observability [8.16] | Elastic
  2. Setup spring boot application on local development environment to send observation data to APM Server
-javaagent:elastic-apm-agent-1.52.0.jar
-Delastic.apm.service_name=demo-service
-Delastic.apm.secret_token=
-Delastic.apm.server_url=http://localhost:8200
-Delastic.apm.environment=ms-test
-Delastic.apm.application_packages=com.example.demo

Provide logs and/or server output (if relevant):

Screenshot 2024-12-09 140904
Screenshot 2024-12-09 1409041590×340 47.2 KB

2

The elastic APM agent skips recording spans for external calls if the current parent span is also an external call. It does this because in general this means that multiple instrumentations (e.g. of a high level and low level HTTP client) are recording the same request.

In your case this does not apply, so the agent wrongly discards the span for the nested HTTP call.

What you can do to workaround this is to add a "normal" span inbetween your HTTP calls: For example add the @CaptureSpan annotation to the callback of your ClientHttpRequestInterceptor which does the HTTP call to fetch the token.
This way, the agent should correctly trace both HTTP calls for you.

3

Thank you @Jonas_Kunz ,

In my case I don't add APM Java agent as a dependency, it's added as a java agent at startup.

Is there any alternative to @CaptureSpan?

Best regards

4

You don't need the agent as a dependency, just the public API dependency. The agent will pickup the annotation regardless of how it is added/started.

Alternatively you can use the trace_methods config option if you don't want to perform any code changes.

1 Like
5

So this is a known issue and there is no plan to include it as a feature in the roadmap.

Thank you for your response @Jonas_Kunz.

Best regards