Full trace with custom integration

Elastic Observability APM
1

Hello,

I am using nodeJS APM agent with a custom integration, for monitoring incomming UDP data in my application.
I am currently splitting my application, with 2 services in charge of different treatment. I was wondering how can I link the transaction generated in both application into a single trace.

Thanks for your help !

2

Welcome to the forum :slight_smile:

Yes, you can link the two transactions together into one trace by ensuring that you pass on the current traceparent from the outgoing service (service A) and read it again in the receiving service (service B). This happens automatically with HTTP based services, but if you use raw TCP or UDP, you need to pass this information along manually.

I assume you use the agent API to manually start a new transaction in service A. During an active transaction, you can always call agent.currentTraceparent, to get the current traceparent string. You need to send this to service B inside your regular UDP request. Then when you start a new transaction in service B using agent.startTransaction(), you need to pass in the traceparent as a childOf option.

Service A example:

agent.startTransaction('my-service-a-transaction')
const traceparent = agent.currentTraceparent

// Somehow send the traceparent as a "header" to service B
sendMetadata(`traceparent: ${traceparent}\n`)

Service B example:

// Somehow read the traceparent from the incoming request
const traceparent = readTraceparentFromUDPPacket()

// Use the traceparent to initialize the new transaction
agent.startTransaction('my-service-b-transaction', { childOf: traceparent })

// continue program as normal...
3

Hi Wa7Son,

Thanks for your help, that is exactly what I was looking for and it works !
Now I have some questions :slight_smile:

  • Is it possible to access labels and custom defined along the transaction in a common place ?
  • When I display the full trace on Kibana, all the traces start at 0ms. Is it the expected behavior ?
    Screenshot%20from%202019-06-18%2014-55-10
    Screenshot from 2019-06-18 14-55-10.png1627×332 16.8 KB
  • Do you advise to use this mechanism inside a single service, where the message could be stored and be dequeud later ? I mean, ending a transaction just before queing the data and beginning a new one at the dequeuing.

Thanks again for your help !

4

I'm not entirely sure what you mean? Could you explain in more details what you would like to see that is not possible in the current UI?

Hmm is should be displayed at the correct time. In your screenshot, is the green diagnostic transaction a child of the blue diagnostic transaction?

I haven't tried that my self, but I don't see why not :slight_smile:

5

I mean that if I add some labels (tags) on my transaction in service A and some others in my transaction in service B, is there a way to display all the labels for the entire trace ? From Elastic point of view, I've got 2 different documents, is it possible to "merge" them in a certain way ?

It should be, if I did well:
Service A (blue one, sad_preprod-eu):

transaction = apm.startTransaction("diagnostic", "HARDWARE");
apmTraceParent = transaction.traceparent;

Service B (green one, localeez-server_preprod-eu):

transaction = apm.startTransaction('diagnostic', 'HARDWARE', { childOf: apmTraceParent })

Let's try :smile:

Thanks !

6

Hi Thanks for using Elastic APM with such an interesting use case.

With respect to this question

When I display the full trace on Kibana, all the traces start at 0ms. Is it the expected behavior ?

Perhaps take a look at this repo, it is Java but I think it is similar, trying to trace across messages and services and get the correct Distributed Trace "Waterfall" result in the UI.

If this this is more your desired behavior I understand there is an issue in the Kibana UI regarding these types of traces starting at 0ms instead of "Following / Waterfall" with the correct timelines.

If so it may be related to

Which appears to be merged as part of this PR

So hopefully that will be part of an soon / upcoming release

Hope that helps a bit...

7

Hi StephenB ! Thanks for the link, It seems to be exactly what I experienced.
I hope that will be part of an upcoming release :slight_smile:

8

Regarding seeing all labels across all transactions in a trace, that's currently not supported and currently not on our roadmap. But I'll take it as a +1 for this feature :slight_smile:

9

@N.Lassimonne if you try my pipeline repo now you should get this result!
Please clone a new copy as I have made some updates.
So now the distributed trace looks like this, I just tested this today with 7.2 the major issue has been fixed, I think there may still be a minor issue but take a look! So you might want to give it a try with your Node.js example

52%20PM

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.