Service Map not linking services

[jhai]

Kibana version: 7.9

Elasticsearch version: 7.9

APM Server version: 7.10

APM Agent language and version: Elastic.Apm.AspNetCore 1.7.1

Browser version: Chrome 87

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Download Page / Nuget

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have a number of Mock services calling REST APIs. A transaction is accurately recorded with each service involved shown in the trace. The documents in the apm* indexes look correct with the required association (traceid, parentid, etc) all looking correct. However, the Service Maps Tab shows 4 unlinked services. I would have expected that a call from one service to another would show as a line with some metrics associated.

Steps to reproduce:

1. Setup Dotnet Core programs/services with APM Agents
2. Capture APM metrics for programs/services with trace id lineage through all
3. Open Service Maps tab and see that services are not linked

Errors in browser console (if relevant): No errors

Provide logs and/or server output (if relevant):

[GregKalapos]

Hi @jhai,

Are these services calling each other with HttpClient? The transaction flow looks a bit strange to me. For example I see the "Event Start" transaction, but then there seems to be no span, below that we immediately see the "HTTP 2xx GET PutData/Create" transaction. There should be an outgoing HTTP request between those. What I wanna get with this is that this may not be related directly to the server map - to me it looks there is another problem here with missing spans.

So if you could give me more info about that app (how do the services call each other? Thing like that), that'd help.

Other thing that could help is logs from these apps on Trace level - would it be possible for you to upload those here?

Greg

[jhai]

Hi @GregKalapos,

Thanks for looking into this. Yes, I am using HttpClient calls to web apis.
I started with minimum explicit instrumentation, which is why there is an absence of spans above.

Its not very clear from the documentation what makes Service Maps work, whether the flow is determined from Services/Transactions/Spans.

Is it correct that I need to manually include Traceparent in the header of the request or should this be added automatically?

Here is HttpClient request from Ws to Ws2:

        var t = Agent.Tracer.CurrentTransaction;
        var s = t.StartSpan("Api1 Span1", "Api Call");
        Uri uri = new Uri("http://localhost:44391/api/HandleData?text=" + text);
        var a = Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData.SerializeToString();
        client.DefaultRequestHeaders.Add("traceparent", a);

        HttpResponseMessage webResponse = await client.GetAsync(uri);
        s.End();

I also not that the Time Spent by Span chart is also blank, not sure if this is related.

Thanks

[GregKalapos]

Hi @jhai,

Below I answer your questions, but before I do so: You can do manual instrumentation for most part and still enable auto instrumentation for outgoing HTTP.

For that, you can simply do this:

Elastic.Apm.Agent.Subscribe(new HttpDiagnosticsSubscriber());

The HttpDiagnosticsSubscriber will turn on auto instrumentation for outgoing HTTP calls. This means each time an outgoing HTTP call happens, the agent will capture it as a span IF there is an active transaction. This means you’ll still need to start your transactions manually (with Tracer.StartTransaction or Tracer.CaptureTransaction), but within a transaction HTTP calls will be automatically captured – so you don’t need any of the span related code (no t.StartSpan, etc.)

The good part about this is that you don’t need to take care of the traceparent header and the whole service map thing will also work – the agent will take care of it automatically.

If you still want to do this manually, you need to do more work. Let me answer your questions, with that it will be hopefully clear what you need to do. But again, I’d rather suggest using HttpDiagnosticsSubscriber and let the agent take care of this.

Its not very clear from the documentation what makes Service Maps work, whether the flow is determined from Services/Transactions/Spans.

Yeah, I agree, we don’t really document this. Service map uses the destination field on spans. These fields are reflected on the public API in C# like this:

transaction.CaptureSpan("foo", "bar", span =>
{
	span.Context.Destination = ...
});

But you don’t really need it – the agent deduces it from other fields. Specifically for HTTP calls it uses the span.Context.Http field to fill the destination field, so span.Context.Http must be set and the agent will do its job to fill necessary fields for the service map.

Is it correct that I need to manually include Traceparent in the header of the request or should this be added automatically?

For the service map you don’t really need that, it only uses the fields I mentioned above. But you’ll still need to manually include the traceparent header in order to make sure the whole thing ends up in a single trace.

But Elastic.Apm.Agent.Subscribe(new HttpDiagnosticsSubscriber()); takes care of both of these things (the destination field and the traceparent header).

I hope this helps.
Greg

[jhai]

Thanks Greg. This is clear to me now and I am able to see the service maps.