Figure out how to link services using manual instrumentation

I have three sample services that I am trying to instrument. Service 1 is linked to Service 2 via rest while Service 2 acts a producer for service 3 (uses AWS SQS). Service 3 is a consumer from SQS.

I have 2 different scenarios which I am trying to understand here.

Here is the code for service 2.

@Get('/sendMessage')
  async sendMessage(@Headers() requestHeaders: any) {
    const activeSpanContext = trace.getSpanContext(context.active());
    return await this.appService.InsertMessage(activeSpanContext);
  }

Scenario 1 :

Code for service 3 :

const recievedMessages = await appService.recieveMessage();
    let activeSpan = trace.getSpan(context.active());
    if (recievedMessages) {
      for (let message of recievedMessages) {
        let contextWeGot = JSON.parse(message['Body'])['ctx'];
        const newCTX = trace.setSpanContext(context.active(), contextWeGot);
        track.startActiveSpan('HTTP GET', {}, newCTX, async (span) => {
          console.log(`Processing Message : ${message['Body']}`);
          span.end();
        });
      }

Service Map :

Scenario 2:

Code for service 3:

const recievedMessages = await appService.recieveMessage();
    let activeSpan = trace.getSpan(context.active());
    if (recievedMessages) {
      for (let message of recievedMessages) {
        let contextWeGot = JSON.parse(message['Body'])['ctx'];
 
        const newCTX = trace.setSpanContext(context.active(), contextWeGot);
 
        // trace.setSpanContext(newCTX, Outerspan);
        track.startActiveSpan('HTTP GET', {}, newCTX, async (span) => {
          console.log(`Processing Message : ${message['Body']}`);
          axios
            .get('https://google.com', {
              headers: {
                ctx: JSON.stringify(trace.getSpanContext(context.active())),
              },
             })
             .then((response) => {
              return response.data;
            });
          span.end();
        });
      }

Service Map:

Question: When I am using just my custom span in service 3, in the service map it shows as a disconnected service, but when I include an axios call (auto-instrumented) in my custom span in service 3, service 3 is then shown as a connected service.

Thanks Jagraj, for the record, this is a case using OpenTelemetry that was initially posted on OpenTelemetry JS repo's Services not linking in Elastic APM service Map · Discussion #2445 · open-telemetry/opentelemetry-js · GitHub

Seems like span attributes played a major role for a span to show up on the map.

'messaging.system': 'AmazonSQS',
'messaging.destination_kind': 'queue',
'messaging.destination': 'our_destination_3',
'messaging.operation': 'process',

Can you also shed some more light here..?

Can you please confirm you use Elastic v7.14 (APM Server, Elasticsearch, and Kibana)?
We have added support for visualization of OpenTelemetry messaging in the Service Map in v7.14 with [OpenTelemetry] Map OpenTelemetry `messaging.*` semantic conventions to Elastic Common Schema · Issue #5094 · elastic/apm-server · GitHub

Please verify the spanKind of the producer, it must be SpanKind Producer (see translate otel messaging.* to ecs by stuartnelson3 · Pull Request #5334 · elastic/apm-server · GitHub)

On the consumer, please set the SpanKind accordingly to Consumer (see translate otel messaging.* to ecs by stuartnelson3 · Pull Request #5334 · elastic/apm-server · GitHub )

Please also fill net.peer.ip

@cyrilleleclerc thanks a lot. It worked like a charm.

1 Like

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.