Capturing information of an entire call cycle(including multiple API consecutive calls)

Hello everyone,
In our project we have a server which works as a router (i.e. routers calls from one server to other appropriate servers).

We have APIs as action API and callback APIs. This means, server1 would call router, which would call server2. After sometime the server2 will call router with the result of the request which router will forward to server1 back.

In short

server 1 => action API on router => server 2
server 2 => callback API on router => server 1

We wanted to figure out if it is possible to capture how many calls are going through this whole cycle and how many of them might be failing?

Please let us know if it is possible to fulfil this requirement using APM. We have tried starting a custom transaction using co.elastic.apm:apm-agent-api, but there seems to be no way to find an existing transaction and stop it in another API call.

Note: We are using multi replica containers, so we might not receive the action and callback on the same pod.

Thank You

Hi @mdubey, welcome to our forum.

Tracking calls from one server to another is what we call "distributed tracing", and is supported without using the API for frameworks that we support : https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html

In your case, you should look for Web frameworks, HTTP clients and RPC frameworks. Only if you are using frameworks/libraries that are not supported for cross-server communication, you should have to use the API.

Thus, the first question is what framework/library you are using to implement those server endpoints and to perform calls to other servers ?

Hi @Sylvain_Juge

Thanks for your reply. Our servers are running using Spring Webflux, which uses Netty under the hood. From this issue, I understand that spring webflux is not supported right now.

However, If I were to go ahead doing it programatically, what approach would you suggest. I have tried including apm-agent-api and taking below steps:

  • Start a transaction in the first api call. Add a label with a request-id
  • Find the transaction with the given request-id(same as previous one) in the subsequent api call and end it. However we are not able to find a way to get an existing transaction.

Please let us know, what would be the right approach to solve this.

In that case, the easiest way to do that is to use the notion of "context" that is provided by Webflux to store the transaction/scope, for example using the strategy described here : https://ndportmann.com/passing-context-with-spring-webflux/ (I've only skimmed over the content, I haven't tried it in practice).

As long as the Transaction object is not ended, it won't be sent and recycled, thus you have to make sure to not leak a reference in the context.

Hi @Sylvain_Juge

We are already using for webflux context for few other requirements (like implementing MDC traceid logging). However in this case the scenario is a bit different. I need to associate a single transaction between two API calls on the same service(maybe be a different pod).

  • On /action I start a transaction (with a unique label)
  • On /callback I find the transaction(using the label) started as part of action call and I end that.

The problem is I don't see a way to find the transaction from APM APIs. I cannot have it in memory because the request can go to different pods of the same service.

Hi @Sylvain_Juge

Did you get a chance to look at above? It would be a great help if we can know how to proceed here.

Thanks

Hello team,

Just bumping this up, for a clarification if our requirement is possible at all to solve using APM. If not please let us know, we will try to find other means to do it.

Thanks