APM service map behavior is different with opentracing

I am using elastic-apm-node agent to create distributed tracing.

Here is my services structures. I have 3 micro services.
producer, kafka, consumer

The call sequence is
producer -> kafka -> consumer

I am not using any auto instrument, I am using the elastic-apm-node-opentracing, and I am using ipc to communicate between the services.
So here is my code to create the tracing.

  • producer
const agent = require('elastic-apm-node').start({serviceName: 'serviceA'});
const tracer = new Tracer(agent);
const tran = tracer.startSpan('request051');
const span = tracer.startSpan('request5101', {references: [opentracing.followsFrom(tran.context())]});
ipc.connectTo('consumer', () => {
    console.log('connected to consumer');
    const headers = {};
    tracer.inject(tran.context(), opentracing.FORMAT_HTT_HEADERS, headers);
    ipc.sendTo('kafka', headers);

    ipc.kafka.on('message', () => {
        console.log('get response from kafka');
        span.finish();
        tran.finish();
    });
});
  • kafka
const agent = require('elastic-apm-node').start({serviceName: 'serviceA'});
const tracer = new Tracer(agent);

ipc.listen((message) => {
   const ctx = tracer.extract(opentracing.FORMAT_HTTP_HEADERS, message);
   const span = tracer.startSpan('kafka 51`, {childOf: ctx});
   ipc.connectTo('consumer', () => {
      ipc.consumer.on('message', () => {
         span.finish();
         ipc.sendTo('producer');
      });
  });
  • consumer
ipc.listen((message) => {
   const ctx = tracer.extract(opentracing.FORMAT_HTTP_HEADERS, message);
   const span = tracer.startSpan('consumer 51`, {childOf: ctx});
   ipc.sendTo('kafka');
   span.finish();
  });

So finally, the distributed tracing timeline looks like this.

Which looks correct, but the Service Map has no links.

And I checked the source code of elastic-apm-node apm-agent-nodejs/http-shared.js at master · elastic/apm-agent-nodejs · GitHub , to display the link of the Service Map correctly, the destionationContext and the httpContext need to be set to the Span (There is no documentation about this part, so I am not sure how to set this destination context correctly to do my own instrument). And this destination context can be only set to the Span object, and a Span need to have an active transaction, so I need to create a Transaction and a Span to display the link. So I am using some hacking way to make the link being displayed.

But since I have to create another transaction for each span, so the timeline looks like this.

You can see in the time line, there is a pair of transaction and span, they have the same duration, so the Transaction is not really useful here, the purpose is just to display the links of the Service Map.

To confirm my understanding about the ServiceMap and the timeline is correct, I am using opentracing library with Jaeger to display the timeline and the service dependency.
And the code is the same with the elastic-apm-node-opentracing code I post above.

The final result is:

So in the Elastic APM, there are two concepts: Transaction and Span, and the Span can not exist without a Transaction, I think this is the key difference, but because of this concept, the behavior different between APM and the opentracing.

So I want to confirm my understanding is correct or not, and is that possible to be totally compatible with opentracing to display service map correctly without additional Transaction.

Thank you.

Thanks for reporting this @jialipassion -- and, by the way, excellent work in getting as far as you did and communicating that back.

You've caught us in the middle of industry changes here. The short version is that elastic-apm-node-opentracing isn't something we're actively supporting -- we're following the industry and putting our efforts behind OpenTelemetry instead. I'm going to provide some background info on our OpenTelemetry support and then talk about the future of the elastic-apm-node-opentracing repo.

With OpenTelemetry there's two options for shipping data to Elastic from your Node.js applications. First, if you have an OpenTelemetry collector receiving data you can export that data from the collector using the elasticexporter from the contrib repository.

Second, if you're running APM Server yourself, we have an experimental feature where APM Server can act-as an OpenTelemetry collector. This will allow you to point the exporters in your Node.js application at APM Server instead of pointing them at an open Telemetry collector. With this option APM Server will honor the span_kind attribute any synthesize a transaction for you. You can see the code that does that here: apm-server/consumer.go at master · elastic/apm-server · GitHub

If you're interested in going to oTel route there's more information here: OpenTelemetry integration | APM Overview [7.12] | Elastic


So while we've moved on from OpenTracing, the apm-agent-nodejs-opentracing repo is still there in order to support folks who find it useful. Updates are limited to community driven PRs at the moment -- we've opened an issue making note of the fact Service Maps won't work -- if anyone reading this has a passion for getting this working we'd love to have that discussion and help figure out what a PR/PRs might look like.

I think that covers it but if you have any follow up questions please don't hesitate to ask. Thanks again for reporting this here!

@alanstorm, thank you very much for the reply.

  1. I am not sure how to make openTelemetry to work with APM, I am using openTelemetry manual instrument, so I assume what I want to do is like the code here opentelemetry-js/tracing.js at main · open-telemetry/opentelemetry-js · GitHub
    So I can create the Span myself and export to the APM. But I didn't find the ElasticSearchAPM exporter in the JS package.

  2. Also about the native collector feature, OpenTelemetry integration | APM Overview [7.12] | Elastic, is that also work with OpenTelemetry manual instrument? So is that possible to provide a nodejs sample?

  3. And if my understanding is correct, if I am using OpenTelemetry, to display the Service Map in the APM correctly, there will only the Span data without the Transaction data, is that right?

Thanks again.