Hello...
I'm trying to create an app that simulates some requests in order to show APM's features. Now I'm working in an attempt to generate a transaction with distributed trace (https://www.elastic.co/guide/en/kibana/master/traces.html#distributed-tracing), but for now I only managed to acumulate sub-transactions inside the mains transaction (which by the way is being generated by APM every time a request to some route in my Node+Express is reached), but I could not get Kibana to show these sub-transactions in a different color in APM's app.
My recipe is the following:
grab the traceparent from the main transaction (the one generated through Express' route)
create a new sub-transaction, which the option "childOf" pointing to this traceparent
create several spans based on this sub-transaction, then close the span
close the sub-transaction
What am I doing wrong?
Thak you!
async generateComplexTransaction(simulationRequest) {
const apmService = new ApmService();
const delayGenerator = new DelayGenerator();
const complexTransaction = simulationRequest.complexTransaction;
const traceparent = apmService.getCurrentTraceparent();
await this.init(simulationRequest);
if (complexTransaction.isDistributed) {
for (let i =0; i < complexTransaction.totalSubTransactions; i++) {
let subTransactionName = `Sub-transaction #${i}`;
let subTransactionType = 'custom';
let subTransactionId = uuid.v4().split('-')[0];
let subTransaction =
apmService.startTransaction(
`${subTransactionName} (${subTransactionId})`,
subTransactionType,
{ childOf: traceparent});
for (let j = 0; j < complexTransaction.totalSpans; j++) {
let randomSpanTypeIndex = util.randomNumber(apmService.SPAN_TYPES().length);
let span = subTransaction.startSpan(`Span #${i}.${j}`, apmService.SPAN_TYPES()[randomSpanTypeIndex]);
await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);
span.end();
await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);
}
subTransaction.end();
}
} else {
for (let j = 0; j < complexTransaction.totalSpans; j++) {
let randomSpanTypeIndex = util.randomNumber(apmService.SPAN_TYPES().length);
let span = apmService.startSpan(`Span #${j}`, apmService.SPAN_TYPES()[randomSpanTypeIndex]);
await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);
span.end();
await delayGenerator.randomDelay(simulationRequest.maxRandomDelay);
}
}
}
If I am reading your code right... they are all the same color because they are all within the same service. (Pretty sure colors are assigned to service names) Distributed trace "crosses" services. If I am reading your code correctly .... technically what you did was not distributed tracing it was just sub transactions and spans within the same service . Distributed tracing implies show the traces between independent services called as part of a larger overall trace. If you made an HTTP call out to another service with a different name, it would show up in another color.
So call out to another HTTP endpoint / Another service and you will then you should see the distributed tracing.. i.e. 2 services with 2 different colors under as single trace.
There is a good blog on Distributed Tracing here :
You can also try these 3 repos that I use it is a React Front End and 2 java services if you want to give it a try. This is a bit of an extension of the blog mentioned above and has a built in "bug" so you can see a long running service.
Hi @stephenb,
first of all, thank you for taking the time to give me some insights on my question.
"technically what you did was not distributed tracing it was just sub transactions and spans within the same service"
Yes, but that's because I was guessing that creating transactions whithn transactions was the way to go to get this distributed trace behaviour.
Just to be clear, what I am trying to build is a simple APM demo application... My goal was to build a show case of APM features, but whithout the overhead of actually using external services (database and rest, for example) and also without actually having to build a real application... I am justing mocking things up (you can see in my code that I have a function to simulate delays).
So, do you think there is anyway I could mock an "external service request" with my app's APM agent so that the APM's server sees it that way?
By the way, how does the agent identifies and organizes these external calls? I mean, how would the NDJSON look like? (I'd like to try to send it manually using the server's REST API)
I'll take a look at the code you suggested, thank's for sharing the links.
Personally I think mocking the data will be much harder than just using the sample I provided. The DB is in memory so looks like a DB without an actual DB, should take about 10min to get running if you run it on localhost.
Or you could just create another Java service for your existing service to call ... if you look at the value estimator service part of the sample that is about as easy http service as can be built you could just take that and refactor it into your own.
Yeah I knew this method (and actually am using it in my code), but from the question of that other thread I (perhaps wrongly) understood that he was trying to achieve the same goal as me, meaning to manually create a remote trace.
I was just curious why the agents don't share the same API interface. If I undestood that correctly, with the startTransactionWithRemoteParent() he was able to trace a remote call inside a parent transaction. As a matter of fact that was I aiming when I tryied to manually break sub-transactions and link them with childOf...
I am still curiousabout how does the agent format a trace so that the server sees it as having a distributed trace... I'll dig deeper into the agent's code to try to find this out.
I found an instrumentation directory, with modules for specific remote services. I guess I need to find which code is using them to find out which headers or something else are set in order to mark as a distributed code...
...but again, you are 100% right when you said that it would be easier just to bring up a simple service and make the call.
1st) I think you are in of a bit of a catch 22 in order to see the distributed trace data you need to run a distributed trace. We do not really publish the data spec / mapping because we are API driven those are the internals / implementation.
Technically you can look at the index template to see the mapping ... It's a bit complex...
And of course the apm-server github repo is public so you could always look at that....
2nd) Why the Agents don't have exact here is some insight I received ..
"...we generally try to make the API's feel native to the language... The main area where we align across agents are config options. We could probably have tried to align more on the specific option you mention here though. But since the other parts of the same API isn't aligned, we didn't see a huge need for it"..
Ok, I could confirm exactly what Stephen said about what makes Kibana color traces with different colors. I manually set one of my sub-transactions with another "service.name":
Connects to a MongoDB database (container at localhost)
Then GETs (with the Axios lib) a JSON from a remote service (JSON Placeholder)
Then creates a collection
Then inserts all users
Then Find all users
At this point I would expect (1) and (2) to be recognized as an external service... But they didn't.
I'll paste my current code below, but it goes without saying that it is still a working in progress...
I think I'm close to get this distributed trace to work, but I remain with the following two doubts:
Why the main transaction ended before the sub-transaction (MongoDB)? Some issue with Promises, or did I do something wrong?
Now I know that what I need is the sub-transaction's "service.name" to be different from the main transaction's. When I startthe agent's client I set "serviceName" and that is the what is used for the main transaction's "service.name", but which component should set the "service.name" for the sub-transactions?
In this scenario should I even be manually creating sub-transactions and spans????
Again, thanks a lot for the help... At the end of this post I'll post a picture that should give an idea of what I am trying to achieve.
async generateDistributedTransaction(simulationRequest) {
const apmService = new ApmService();
const delayGenerator = new DelayGenerator();
const distributedTransaction = simulationRequest.distributedTransaction;
const currentTransaction = apmService.getCurrentTransaction();
const currentTraceparent = apmService.getCurrentTraceparent();
await this.init(simulationRequest, false);
let usersTransaction = apmService.startTransaction('MongoDB', 'db.mongodb.connect', { childOf: currentTraceparent });
let connectToMongoDbSpan = usersTransaction.startSpan('Connect to MongoDB');
const url = 'mongodb://root:password@localhost:27017';
mongo.connect(url, (err, client) => {
if (err) {
connectToMongoDbSpan.end();
usersTransaction.end();
console.error(err)
return
}
connectToMongoDbSpan.end();
let getUsersJsonSpan = usersTransaction.startSpan('Get Users JSON');
axios.get('https://jsonplaceholder.typicode.com/users')
.then(response => {
getUsersJsonSpan.end();
const data = response.data;
let createUsersCollectionSpan = usersTransaction.startSpan('Create Users collection');
const db = client.db('apm-demo');
const usersCollection = db.collection('users');
createUsersCollectionSpan.end();
let insertIntoUsersCollectionSpan = usersTransaction.startSpan('Insert into Users collection');
usersCollection.insertMany(data, (err, result) => {
insertIntoUsersCollectionSpan.end();
if (err) {
throw err;
}
let findUsersSpan = usersTransaction.startSpan('Find Users', 'db.mongodb.query');
usersCollection.find().toArray((err, items) => {
findUsersSpan.end();
usersTransaction.end();
console.log(`Got ${items.length} items`);
});
});
})
.catch(error => {
usersTransaction.end();
console.log(error);
});
});
}
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
