APM distributed trace: how to?

ffknob · August 3, 2019, 5:39pm

Hello...
I'm trying to create an app that simulates some requests in order to show APM's features. Now I'm working in an attempt to generate a transaction with distributed trace (https://www.elastic.co/guide/en/kibana/master/traces.html#distributed-tracing), but for now I only managed to acumulate sub-transactions inside the mains transaction (which by the way is being generated by APM every time a request to some route in my Node+Express is reached), but I could not get Kibana to show these sub-transactions in a different color in APM's app.

My recipe is the following:

grab the traceparent from the main transaction (the one generated through Express' route)
create a new sub-transaction, which the option "childOf" pointing to this traceparent
create several spans based on this sub-transaction, then close the span
close the sub-transaction

What am I doing wrong?

Thak you!

async generateComplexTransaction(simulationRequest) {
	const apmService = new ApmService();
	const delayGenerator = new DelayGenerator();

	const complexTransaction = simulationRequest.complexTransaction;
	const traceparent = apmService.getCurrentTraceparent();

	await this.init(simulationRequest);

	if (complexTransaction.isDistributed) {
	    for (let i =0; i < complexTransaction.totalSubTransactions; i++) {
		let subTransactionName = `Sub-transaction #${i}`;
		let subTransactionType = 'custom';
		let subTransactionId = uuid.v4().split('-')[0];
		let subTransaction = 
		    apmService.startTransaction(
		        `${subTransactionName} (${subTransactionId})`,
		        subTransactionType,
		        { childOf:  traceparent});

		for (let j = 0; j < complexTransaction.totalSpans; j++) {
		    let randomSpanTypeIndex = util.randomNumber(apmService.SPAN_TYPES().length);
		    let span = subTransaction.startSpan(`Span #${i}.${j}`, apmService.SPAN_TYPES()[randomSpanTypeIndex]);

		    await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);

		    span.end();

		    await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);
		}

		subTransaction.end();
	    }
	} else {
	    for (let j = 0; j < complexTransaction.totalSpans; j++) {
		let randomSpanTypeIndex = util.randomNumber(apmService.SPAN_TYPES().length);
		let span = apmService.startSpan(`Span #${j}`, apmService.SPAN_TYPES()[randomSpanTypeIndex]);

		await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);

		span.end();

		await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);
	    }
	}
}

Result:

stephenb · August 3, 2019, 8:07pm

Hi @ffknob you are well on your way.

If I am reading your code right... they are all the same color because they are all within the same service. (Pretty sure colors are assigned to service names) Distributed trace "crosses" services. If I am reading your code correctly .... technically what you did was not distributed tracing it was just sub transactions and spans within the same service . Distributed tracing implies show the traces between independent services called as part of a larger overall trace. If you made an HTTP call out to another service with a different name, it would show up in another color.

So call out to another HTTP endpoint / Another service and you will then you should see the distributed tracing.. i.e. 2 services with 2 different colors under as single trace.

There is a good blog on Distributed Tracing here :

You can also try these 3 repos that I use it is a React Front End and 2 java services if you want to give it a try. This is a bit of an extension of the blog mentioned above and has a built in "bug" so you can see a long running service.

This should give you a trace like.

ffknob · August 4, 2019, 9:23am

Hi @stephenb,
first of all, thank you for taking the time to give me some insights on my question.

"technically what you did was not distributed tracing it was just sub transactions and spans within the same service"

Yes, but that's because I was guessing that creating transactions whithn transactions was the way to go to get this distributed trace behaviour.

Just to be clear, what I am trying to build is a simple APM demo application... My goal was to build a show case of APM features, but whithout the overhead of actually using external services (database and rest, for example) and also without actually having to build a real application... I am justing mocking things up (you can see in my code that I have a function to simulate delays).

So, do you think there is anyway I could mock an "external service request" with my app's APM agent so that the APM's server sees it that way?

By the way, how does the agent identifies and organizes these external calls? I mean, how would the NDJSON look like? (I'd like to try to send it manually using the server's REST API)

I'll take a look at the code you suggested, thank's for sharing the links.

Once again, thank you for your help!

stephenb · August 4, 2019, 2:22pm

Personally I think mocking the data will be much harder than just using the sample I provided. The DB is in memory so looks like a DB without an actual DB, should take about 10min to get running if you run it on localhost.

Or you could just create another Java service for your existing service to call ... if you look at the value estimator service part of the sample that is about as easy http service as can be built you could just take that and refactor it into your own.

Good luck let us know.

ffknob · August 5, 2019, 4:25pm

I'll definitely give it a try!
Thank's for the help!

ffknob · August 7, 2019, 11:40pm

I was wondering why the Node.js agent doesn't seem to have the startTransactionWithRemoteParent() method like the Java one...

stephenb · August 8, 2019, 3:19am

Here ?

`apm.startTransaction([name][, type][, options])`

Added in: v0.1.0

name <string> The name of the transaction. You can always set this later via transaction.name or apm.setTransactionName() . Default: unnamed
type <string> The type of transaction. You can always set this later via transaction.type . Default: custom
options <Object> The following options are supported:
.....
childOf <string> The traceparent header received from a remote service.

ffknob · August 8, 2019, 11:40am

Yeah I knew this method (and actually am using it in my code), but from the question of that other thread I (perhaps wrongly) understood that he was trying to achieve the same goal as me, meaning to manually create a remote trace.

I was just curious why the agents don't share the same API interface. If I undestood that correctly, with the startTransactionWithRemoteParent() he was able to trace a remote call inside a parent transaction. As a matter of fact that was I aiming when I tryied to manually break sub-transactions and link them with childOf...

I am still curiousabout how does the agent format a trace so that the server sees it as having a distributed trace... I'll dig deeper into the agent's code to try to find this out.

I found an instrumentation directory, with modules for specific remote services. I guess I need to find which code is using them to find out which headers or something else are set in order to mark as a distributed code...

...but again, you are 100% right when you said that it would be easier just to bring up a simple service and make the call.

stephenb · August 8, 2019, 10:15pm

Hi @ffknob

1st) I think you are in of a bit of a catch 22 in order to see the distributed trace data you need to run a distributed trace. We do not really publish the data spec / mapping because we are API driven those are the internals / implementation.

Technically you can look at the index template to see the mapping ... It's a bit complex...

And of course the apm-server github repo is public so you could always look at that....

2nd) Why the Agents don't have exact here is some insight I received ..

"...we generally try to make the API's feel native to the language... The main area where we align across agents are config options. We could probably have tried to align more on the specific option you mention here though. But since the other parts of the same API isn't aligned, we didn't see a huge need for it"..

ffknob · August 13, 2019, 12:53pm

So I completly abandoned my "mock all the things" approach... I took a shot with http + mongodb:

Grab some data from https://jsonplaceholder.typicode.com/ (using the axios node lib to make a request and receive a json array)
I then create a mongodb collection and insert all the array in it (using mongodb node lib)
After I do that I immediatly run a find() to grab all the documents back...

With that I expected to finally see a distributed trace (a axios/http request span, then mongodb/insert span and a mongob/find span).

I guess I got some basic understanding about APM + Distributed trace very wrong...

So I'll here leave some very basic doubts that I currently have (consider that I am using Node+Express):

The APM agent creates a transaction for every Express route reached. Do I have to manually create spans inside the code that is executed for th route?
Does the agent identifies by itself the many possible different instrumentation modules (mongodb, elasticsearch, redis, http, ...) calls?

Thank you

ffknob · August 18, 2019, 9:17pm

For those interested in learning more about APM, just found an awesome talk from Thomas Watson at the Elastic Copenhagen Meetup from last year...

Here is the link:

ffknob · August 19, 2019, 2:04pm

Ok, I could confirm exactly what Stephen said about what makes Kibana color traces with different colors. I manually set one of my sub-transactions with another "service.name":

POST apm-7.2.0-transaction-2019.08.19/_update/guEcqmwBgUxECPnRvxm4
{
  "doc": {
    "service": {
      "name": "external-db"
    }
  }
}

And here is how it looks in Kibana:

What my code does now is:

Connects to a MongoDB database (container at localhost)
Then GETs (with the Axios lib) a JSON from a remote service (JSON Placeholder)
Then creates a collection
Then inserts all users
Then Find all users

At this point I would expect (1) and (2) to be recognized as an external service... But they didn't.
I'll paste my current code below, but it goes without saying that it is still a working in progress...

I think I'm close to get this distributed trace to work, but I remain with the following two doubts:

Why the main transaction ended before the sub-transaction (MongoDB)? Some issue with Promises, or did I do something wrong?
Now I know that what I need is the sub-transaction's "service.name" to be different from the main transaction's. When I startthe agent's client I set "serviceName" and that is the what is used for the main transaction's "service.name", but which component should set the "service.name" for the sub-transactions?
In this scenario should I even be manually creating sub-transactions and spans????

Again, thanks a lot for the help... At the end of this post I'll post a picture that should give an idea of what I am trying to achieve.

async generateDistributedTransaction(simulationRequest) {
        const apmService = new ApmService();
        const delayGenerator = new DelayGenerator();

        const distributedTransaction = simulationRequest.distributedTransaction;
        const currentTransaction = apmService.getCurrentTransaction();
        const currentTraceparent = apmService.getCurrentTraceparent();

        await this.init(simulationRequest, false);

        let usersTransaction = apmService.startTransaction('MongoDB', 'db.mongodb.connect', { childOf: currentTraceparent });

        let connectToMongoDbSpan = usersTransaction.startSpan('Connect to MongoDB');

        const url = 'mongodb://root:password@localhost:27017';
        mongo.connect(url, (err, client) => {
            if (err) {
                connectToMongoDbSpan.end();
                usersTransaction.end();

                console.error(err)
                return
            }

            connectToMongoDbSpan.end();

            let getUsersJsonSpan = usersTransaction.startSpan('Get Users JSON');

            axios.get('https://jsonplaceholder.typicode.com/users')
                .then(response => {
                    getUsersJsonSpan.end();

                    const data = response.data;

                    let createUsersCollectionSpan = usersTransaction.startSpan('Create Users collection');

                    const db = client.db('apm-demo');        
                    const usersCollection = db.collection('users');
        
                    createUsersCollectionSpan.end();

                    let insertIntoUsersCollectionSpan = usersTransaction.startSpan('Insert into Users collection');

                    usersCollection.insertMany(data, (err, result) => {
                        insertIntoUsersCollectionSpan.end();

                        if (err) {
                            throw err;
                        }

                        let findUsersSpan = usersTransaction.startSpan('Find Users', 'db.mongodb.query');

                        usersCollection.find().toArray((err, items) => {
                            findUsersSpan.end();
                            usersTransaction.end();

                            console.log(`Got ${items.length} items`);
                        });
                    });
                })
                .catch(error => {
                    usersTransaction.end();

                    console.log(error);
                });
          });
    }

My app's frontend:

system · September 9, 2019, 10:04am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
APM transaction propogation does not shown in Kibana Elastic Observability	3	248	November 4, 2022
Debugging information for JAVA agent APM	27	1793	January 24, 2019
Manually propagating a distributed tracing context from third party http client (.net) APM docker , dotnet , nodejs , language-clients	11	1059	October 26, 2022
Full trace with custom integration APM nodejs	9	1147	July 16, 2019
Distributed tracing with custom transactions (Node.js agent) APM	6	565	February 13, 2019

APM distributed trace: how to?

apm.startTransaction([name][, type][, options])

Related topics

`apm.startTransaction([name][, type][, options])`