Child transaction starts before parent transaction

[Slavik_Muzychyshyn]

We need to measure the latency of the request

The request passes through 2 microservices

The business task is to measure the time of processing on each microservice and also measure the time of network between microservices

We use this article to achieve distributed tracing

As you can see for this picture there are:

• Main root transaction - poker-frontend-game-actions-call-latency
• Child transaction - poker-frontend-game-actions-call-latency-games-ws

So we:

• Create transaction (first microservice)
• Inject transparent id into headers (first microservice)
• Send request (first microservice)
• Get request (second microservice)
• Get transparent id from headers and create a transaction from parentId

So the problem is that the start time of child transaction is faster than parent transaction (you can see it on pictures below)

Any ideas? how can I debug this?

[felixbarny (Felix Barnsteiner)]

A potential cause is that the clocks run a bit differently on your two services. Try syncing the clocks via NTP to get the clocks more aligned.
But with distributed systems, it's impossible to have fully in-sync clocks.

[Slavik_Muzychyshyn]

But one transaction we send from the client browser (rum-js) and another from the backend server (java). How we can sync them?

[felixbarny (Felix Barnsteiner)]

The start timestamp for RUM transactions is determined server arrival time - duration. It's just a best guess rather than an exact time. That's because the time of the clients can't be trusted. You can't run NTP on all of your user's desktops and mobile phones

If I recall correctly, the UI does some compensation when lining up frontend and backend transaction to ensure the backend transaction does not appear to be starting before the frontend transaction. When looking at the exact timestamps, there's not really a way to hide the fact that clock syncing of distributed systems that you don't control is an unsolvable problem.

Often, looking at the span waterfall graph with a lens of knowing that there may be a time offset for different services is good enough to infer what's going on. If you don't feel that's the case for you, please elaborate on the reasons why.