APM service map behavior is different with opentracing

[jialipassion (LI JIA)]

I am using elastic-apm-node agent to create distributed tracing.

Here is my services structures. I have 3 micro services.
producer, kafka, consumer

The call sequence is
producer -> kafka -> consumer

I am not using any auto instrument, I am using the elastic-apm-node-opentracing, and I am using ipc to communicate between the services.
So here is my code to create the tracing.

• producer

const agent = require('elastic-apm-node').start({serviceName: 'serviceA'});
const tracer = new Tracer(agent);
const tran = tracer.startSpan('request051');
const span = tracer.startSpan('request5101', {references: [opentracing.followsFrom(tran.context())]});
ipc.connectTo('consumer', () => {
    console.log('connected to consumer');
    const headers = {};
    tracer.inject(tran.context(), opentracing.FORMAT_HTT_HEADERS, headers);
    ipc.sendTo('kafka', headers);

    ipc.kafka.on('message', () => {
        console.log('get response from kafka');
        span.finish();
        tran.finish();
    });
});

• kafka

const agent = require('elastic-apm-node').start({serviceName: 'serviceA'});
const tracer = new Tracer(agent);

ipc.listen((message) => {
   const ctx = tracer.extract(opentracing.FORMAT_HTTP_HEADERS, message);
   const span = tracer.startSpan('kafka 51`, {childOf: ctx});
   ipc.connectTo('consumer', () => {
      ipc.consumer.on('message', () => {
         span.finish();
         ipc.sendTo('producer');
      });
  });

• consumer

ipc.listen((message) => {
   const ctx = tracer.extract(opentracing.FORMAT_HTTP_HEADERS, message);
   const span = tracer.startSpan('consumer 51`, {childOf: ctx});
   ipc.sendTo('kafka');
   span.finish();
  });

So finally, the distributed tracing timeline looks like this.

Which looks correct, but the Service Map has no links.

And I checked the source code of elastic-apm-node apm-agent-nodejs/http-shared.js at master · elastic/apm-agent-nodejs · GitHub , to display the link of the Service Map correctly, the destionationContext and the httpContext need to be set to the Span (There is no documentation about this part, so I am not sure how to set this destination context correctly to do my own instrument). And this destination context can be only set to the Span object, and a Span need to have an active transaction, so I need to create a Transaction and a Span to display the link. So I am using some hacking way to make the link being displayed.

But since I have to create another transaction for each span, so the timeline looks like this.

You can see in the time line, there is a pair of transaction and span, they have the same duration, so the Transaction is not really useful here, the purpose is just to display the links of the Service Map.

To confirm my understanding about the ServiceMap and the timeline is correct, I am using opentracing library with Jaeger to display the timeline and the service dependency.
And the code is the same with the elastic-apm-node-opentracing code I post above.

The final result is:

So in the Elastic APM, there are two concepts: Transaction and Span, and the Span can not exist without a Transaction, I think this is the key difference, but because of this concept, the behavior different between APM and the opentracing.

So I want to confirm my understanding is correct or not, and is that possible to be totally compatible with opentracing to display service map correctly without additional Transaction.

Thank you.

[alanstorm]

Thanks for reporting this @jialipassion -- and, by the way, excellent work in getting as far as you did and communicating that back.

You've caught us in the middle of industry changes here. The short version is that elastic-apm-node-opentracing isn't something we're actively supporting -- we're following the industry and putting our efforts behind OpenTelemetry instead. I'm going to provide some background info on our OpenTelemetry support and then talk about the future of the elastic-apm-node-opentracing repo.

With OpenTelemetry there's two options for shipping data to Elastic from your Node.js applications. First, if you have an OpenTelemetry collector receiving data you can export that data from the collector using the elasticexporter from the contrib repository.

Second, if you're running APM Server yourself, we have an experimental feature where APM Server can act-as an OpenTelemetry collector. This will allow you to point the exporters in your Node.js application at APM Server instead of pointing them at an open Telemetry collector. With this option APM Server will honor the span_kind attribute any synthesize a transaction for you. You can see the code that does that here: apm-server/consumer.go at master · elastic/apm-server · GitHub

If you're interested in going to oTel route there's more information here: OpenTelemetry integration | APM Overview [7.12] | Elastic

---

So while we've moved on from OpenTracing, the apm-agent-nodejs-opentracing repo is still there in order to support folks who find it useful. Updates are limited to community driven PRs at the moment -- we've opened an issue making note of the fact Service Maps won't work -- if anyone reading this has a passion for getting this working we'd love to have that discussion and help figure out what a PR/PRs might look like.

I think that covers it but if you have any follow up questions please don't hesitate to ask. Thanks again for reporting this here!

[jialipassion (LI JIA)]

@alanstorm, thank you very much for the reply.

1. I am not sure how to make openTelemetry to work with APM, I am using openTelemetry manual instrument, so I assume what I want to do is like the code here opentelemetry-js/tracing.js at main · open-telemetry/opentelemetry-js · GitHub
   So I can create the Span myself and export to the APM. But I didn't find the ElasticSearchAPM exporter in the JS package.

2. Also about the native collector feature, OpenTelemetry integration | APM Overview [7.12] | Elastic, is that also work with OpenTelemetry manual instrument? So is that possible to provide a nodejs sample?

3. And if my understanding is correct, if I am using OpenTelemetry, to display the Service Map in the APM correctly, there will only the Span data without the Transaction data, is that right?

Thanks again.

[alanstorm]

I am not sure how to make openTelemetry to work ... I didn't find the ElasticSearchAPM exporter

We hear that -- OpenTelemetry and exporters can be a little tricky to get your head around at first. I'm not exactly sure which part you're having trouble with -- let me know if this gets you pointed in the right direction.

An exporter allows you to get information out of a system that's instrumented with OpenTelemetry. Whether it's manually or automatically instrumented shouldn't matter. If you're generating spans with OpenTelemetry those spans will be handed off to an exporter. You can test this by using the OpenTelemetry ConsoleSpanExporter, which "exports" spans by printing them to the console. This (unaffiliated with Elastic) tutorial covers using the ConsoleSpanExporter with a simple express based service.

In a real world system an exporter won't send spans to the console. It will send them to whatever span storage system you're using. Broadly speaking there's two strategies exporters take to send data elsewhere. The first is to have an exporter send that data directly to the span storage system. The second is to have an exporter send that data to an OpenTelemetry collector, and the collector will send that data to the span storage system.

Right now Elastic is opting for strategy two. We don't have an exporter that sends data directly to a span storage system. Instead, you'll configure OpenTelemetry with a collector exporter, and configure that exporter with a collector URL. With this configuration in place, your spans will be sent to the OpenTelemetry collector.

In other words, the exporter you'll want to use is the CollectorTraceExporter exported by @opentelemetry/exporter-collector.

The Collector

Implicit in all this is that you'll need to be running an OpenTelemetry collector instance somewhere. A collector exposes an endpoint/url, and you point your CollectorTraceExporter at this URL.

There's a choice you'll need to make here. You can run the OpenTelemetry collector instance yourself, or you can have APM Server can act as an OpenTelemetry collector.

If you're having APM Server act as the OpenTelemetry collector, then all you need to do is point your CollectorTraceExporter at APM Server. The guide information for this is here. The docs are specific to java, but should be relatively straightforward to convert to using with the @opentelemetry/exporter-collector/CollectorTraceExporter. If you run into issues let us know.

If you're running an OpenTelemetry collector instance yourself, then you need to take additional steps to move data from the OpenTelemetry collector to your span storage system. Those additional steps are to use a collector exporter. Yes, another exporter.

A collector exporter moves spans from a collector to your span storage system. This is the contrib exporter I mentioned in my previous post. The guide information for doing this is here.

Transaction-less Future

Re:

if I am using OpenTelemetry, to display the Service Map in the APM correctly, there will only the Span data without the Transaction data, is that right?

That is correct. OpenTelemetry's data model is transactionless -- it uses only spans. The limitation you ran into earlier was that the Node.js Agent requires a transaction to be present in order to start a span. Our old OpenTracing solution is based on the agent, so it shared this limitation. With OpenTelemetry, the Node.js Agent isn't in the mix, so this limitation doesn't exist.

Finally -- apologies for the wall of text. OpenTelemetry is am ambitious project. If you have additional questions please don't hesitate to keep this conversation going.