# Full trace with custom integration **URL:** \ **Category:** APM\ **Tags:** nodejs\ **Created:** [June 17, 2019, 5:58am UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996 "2019-06-17T05:58:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![N.Lassimonne](https://avatars.discourse-cdn.com/v4/letter/n/43a26b/32.png) [@N.Lassimonne](https://discuss.elastic.co/u/N.Lassimonne)\ **Post date:** [June 17, 2019, 5:58am UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/1 "2019-06-17T05:58:55Z")
Hello, I am using nodeJS APM agent with a custom integration, for monitoring incomming UDP data in my application. I am currently splitting my application, with 2 services in charge of different treatment. I was wondering how can I link the transaction generated in both application into a single trace. Thanks for your help ! ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [June 17, 2019, 7:57am UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/2 "2019-06-17T07:57:51Z")
Welcome to the forum πŸ™‚ Yes, you can link the two transactions together into _one_ trace by ensuring that you pass on the current `traceparent` from the outgoing service (service A) and read it again in the receiving service (service B). This happens automatically with HTTP based services, but if you use raw TCP or UDP, you need to pass this information along manually. I assume you use the agent API to manually start a new transaction in service A. During an active transaction, you can always call [`agent.currentTraceparent`](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-current-traceparent), to get the current `traceparent` string. You need to send this to service B inside your regular UDP request. Then when you start a new transaction in service B using [`agent.startTransaction()`](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-start-transaction), you need to pass in the `traceparent` as a `childOf` option. Service A example: ```auto agent.startTransaction('my-service-a-transaction') const traceparent = agent.currentTraceparent // Somehow send the traceparent as a "header" to service B sendMetadata(`traceparent: ${traceparent}\n`) ``` Service B example: ```auto // Somehow read the traceparent from the incoming request const traceparent = readTraceparentFromUDPPacket() // Use the traceparent to initialize the new transaction agent.startTransaction('my-service-b-transaction', { childOf: traceparent }) // continue program as normal... ``` ---
**Author:** ![N.Lassimonne](https://avatars.discourse-cdn.com/v4/letter/n/43a26b/32.png) [@N.Lassimonne](https://discuss.elastic.co/u/N.Lassimonne)\ **Post date:** [June 18, 2019, 11:04am UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/3 "2019-06-18T11:04:33Z")
Hi Wa7Son, Thanks for your help, that is exactly what I was looking for and it works ! Now I have some questions πŸ™‚ - Is it possible to access labels and custom defined along the transaction in a common place ? - When I display the full trace on Kibana, all the traces start at 0ms. Is it the expected behavior ? ![Screenshot%20from%202019-06-18%2014-55-10](https://us1.discourse-cdn.com/elastic/original/3X/5/7/5799404d9813edc1a0d28446a3e3f81e5d60fb99.png) - Do you advise to use this mechanism inside a single service, where the message could be stored and be dequeud later ? I mean, ending a transaction just before queing the data and beginning a new one at the dequeuing. Thanks again for your help ! ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [June 18, 2019, 1:44pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/4 "2019-06-18T13:44:06Z")
> [@N.Lassimonne](#): > > - Is it possible to access labels and custom defined along the transaction in a common place ? I'm not entirely sure what you mean? Could you explain in more details what you would like to see that is not possible in the current UI? > [@N.Lassimonne](#): > > - When I display the full trace on Kibana, all the traces start at 0ms. Is it the expected behavior ? Hmm is should be displayed at the correct time. In your screenshot, is the green `diagnostic` transaction a child of the blue `diagnostic` transaction? > [@N.Lassimonne](#): > > - Do you advise to use this mechanism inside a single service, where the message could be stored and be dequeud later ? I mean, ending a transaction just before queing the data and beginning a new one at the dequeuing. I haven't tried that my self, but I don't see why not πŸ™‚ ---
**Author:** ![N.Lassimonne](https://avatars.discourse-cdn.com/v4/letter/n/43a26b/32.png) [@N.Lassimonne](https://discuss.elastic.co/u/N.Lassimonne)\ **Post date:** [June 18, 2019, 2:07pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/5 "2019-06-18T14:07:32Z")
> [@wa7son](#): > > I'm not entirely sure what you mean? Could you explain in more details what you would like to see that is not possible in the current UI? I mean that if I add some labels (tags) on my transaction in service A and some others in my transaction in service B, is there a way to display all the labels for the entire trace ? From Elastic point of view, I've got 2 different documents, is it possible to "merge" them in a certain way ? > [@wa7son](#): > > Hmm is should be displayed at the correct time. In your screenshot, is the green `diagnostic` transaction a child of the blue `diagnostic` transaction? It should be, if I did well: Service A (blue one, sad\_preprod-eu): ```javascript transaction = apm.startTransaction("diagnostic", "HARDWARE"); apmTraceParent = transaction.traceparent; ``` Service B (green one, localeez-server\_preprod-eu): ```javascript transaction = apm.startTransaction('diagnostic', 'HARDWARE', { childOf: apmTraceParent }) ``` > [@wa7son](#): > > I haven't tried that my self, but I don't see why not πŸ™‚ Let's try πŸ˜„ Thanks ! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 18, 2019, 2:32pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/6 "2019-06-18T14:32:53Z")
Hi Thanks for using Elastic APM with such an interesting use case. With respect to this question > When I display the full trace on Kibana, all the traces start at 0ms. Is it the expected behavior ? Perhaps take a look at this repo, it is Java but I think it is similar, trying to trace across messages and services and get the correct Distributed Trace "Waterfall" result in the UI. > **[GitHub - bvader/pipelineapmexample: Example of pipeline that uses APM...](https://github.com/bvader/pipelineapmexample)** > > Example of pipeline that uses APM pefrormance metrics - GitHub - bvader/pipelineapmexample: Example of pipeline that uses APM pefrormance metrics If this this is more your desired behavior I understand there is an issue in the Kibana UI regarding these types of traces starting at 0ms instead of "Following / Waterfall" with the correct timelines. If so it may be related to > > > Background: https://github.com/elastic/kibana/issues/24072 > > This is how we cur…rently do time shifting: > https://github.com/elastic/kibana/blob/af3594f6c3663af42058c00bcc9ffb9dc931491b/x-pack/plugins/apm/public/components/app/TransactionDetails/Transaction/WaterfallContainer/Waterfall/waterfall\_helpers/waterfall\_helpers.ts#L123-L159 > > Time shifting in its current form may be causing more harm than good, especially for \[users using async message queues\](https://discuss.elastic.co/t/apm-transactions-displayed-in-wrong-time-order/178406). > > Solution could be to only time shift when child start before parent. Not if the child starts after parent has ended. Which appears to be merged as part of this PR > > > Closes #35800 So hopefully that will be part of an soon / upcoming release Hope that helps a bit... ---
**Author:** ![N.Lassimonne](https://avatars.discourse-cdn.com/v4/letter/n/43a26b/32.png) [@N.Lassimonne](https://discuss.elastic.co/u/N.Lassimonne)\ **Post date:** [June 18, 2019, 2:44pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/7 "2019-06-18T14:44:46Z")
Hi StephenB ! Thanks for the link, It seems to be exactly what I experienced. I hope that will be part of an upcoming release πŸ™‚ ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [June 18, 2019, 6:10pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/8 "2019-06-18T18:10:37Z")
Regarding seeing all labels across all transactions in a trace, that's currently not supported and currently not on our roadmap. But I'll take it as a +1 for this feature πŸ™‚ ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 25, 2019, 11:11pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/9 "2019-06-25T23:11:49Z")
@N.Lassimonne if you try my [pipeline repo](https://github.com/bvader/pipelineapmexample) now you should get this result! Please clone a new copy as I have made some updates. So now the distributed trace looks like this, I just tested this today with 7.2 the major issue has been fixed, I think there may still be a minor issue but take a look! So you might want to give it a try with your Node.js example ![52%20PM](https://us1.discourse-cdn.com/elastic/original/3X/9/7/97bb1c764a8745cd9abdd73794b8ed4764ae791c.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 16, 2019, 7:12pm UTC](https://discuss.elastic.co/t/full-trace-with-custom-integration/185996/10 "2019-07-16T19:12:00Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.