.Net APM Traces Incomplete When Deployed

Kibana version: 7.17..0

Elasticsearch version: 7.17.0

APM Server version: 7.17.15

APM Agent language and version: .Net Core 6.0

Browser version: Chrome 119.0.6045.160

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Download page

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? Using reverse proxies in front of Elastic/Kibana/APM

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have two ASP .Net Core applications, both running .Net Core APM Agents:

  • CIS

When running the applications on my local machine in IIS Express, APM entries are being delivered to Kibana, correlated as expected, and the Traces show the complete history including entries from both CIS and CIS API. Here's a specific instance of this for a particular request originating in CIS:


This trace consists of 4 items:
A) Initial web request to CIS
B) Outgoing HTTP request from CIS to CIS API
C) Incoming request to CIS API
D) Database operation performed by the API

Once deploying the same code to the server (running IIS), however, a few things happen:

  1. The Trace only shows a single entry from CIS ('A' from above)

  2. Querying the Logs for records with the same trace.id as in the above record, however:
    a. There are entry records being captured for the CIS API incoming request and database operation (records 'C' and 'D' from above)
    b. No event record exists for the outgoing HTTP request event from CIS to the CIS API (record 'B' from above)

My questions are:

  1. Why are my Traces showing as incomplete in Kibana, despite having correlated entries with the same trace.id?
  2. Why are not all outgoing HTTP requests from CIS being captured?

Things I've tried so far:

  1. The same missing trace entry behavior also occurs if I run CIS locally, hitting the deployed version of CIS API.
  2. In the deployed instances, CIS does indeed capture a few outgoing HTTP requests (to other APM tracked services), but the majority of them are not captured (including all to the API).
  3. Examining requests from a Locally running CIS to both the Local and Deployed versions of the API, there are inconsistent inclusions of the 'elastic-apm-traceparent' request header. The request to the Local API includes an 'elastic-apm-traceparent' header, however the request to the Deployed API does not.

Steps to reproduce:

  1. Both applications are configured using


  1. Calls to the API are made using using the following code:
public async Task<T> GetStats<T>(string query_params)
    var endpoint = $"Stats?{query_params}";

    var request = new HttpRequestMessage(HttpMethod.Get, endpoint);
    var client = _clientFactory.CreateClient(HttpClients.CISApi);
    var response = await client.SendAsync(request);

    var json = await response.Content.ReadAsStringAsync();
    var data = JsonConvert.DeserializeObject<T>(json, new JsonSerializerSettings
        NullValueHandling = NullValueHandling.Ignore

    return data;

Errors in browser console (if relevant): N/A

Provide logs and/or server output (if relevant): N/A

Happy to provide more info, if needed. Thanks!

I was discussing this issue with a co-worker today and he mentioned something that might be a factor in this. When deployed, our CIS API is running as a virtual directory under the CIS application, like this:

CIS: cis
CIS API: cis/api

Is it possible that this is the reason why it's not capturing the outgoing HTTP request to the API because it's not aware that it's actually an external call (because the source and destination have the same base domain)? Could that also explain why the CIS and API calls are not being correlated? Is there a way to explicitly tell the APM to treat those calls as 'external'?

Ok, more research findings and a solution! I downloaded the source for the .Net Core APM library and was able to debug the HTTP Diagnostic Listener and trace the issue to this line of code in the Elastic.Apm project, HttpDiagnosticListenerImplBase class:

Line 337:
private bool IsRequestFilteredOut(Uri requestUri) => ApmAgent.Configuration.ServerUrl.IsBaseOf(requestUri);

What's happening is that because my APM server is also set up as a virtual directory on the same server via a reverse proxy (hosted at cis/apm), the base url of both the API and the APM server are the same. This result is that request is being determined to be a call to the APM server, and therefore filtered out of the HTTP diagnostics listener.

I'm not sure of the best way to resolve this issue, however one option is to use a different method of determining if the request is going to the APM server by comparing the start of the request urls:

private bool IsRequestFilteredOut(Uri requestUri) => requestUri?.ToString().StartsWith(ApmAgent.Configuration.ServerUrl?.ToString(), StringComparison.InvariantCultureIgnoreCase) ?? false;

Making this change results in the missing HTTP request entry now being included and also the full Trace showing in Kibana. I'll open an issue in github for this.

One final finding here: I was able to avoid having to update the code by simply adding a trailing slash '/' to my APM server url. This is enough for .Net to consider a request to 'cis/apm/' and 'cis/api/' as having different base urls. Without this update, a request to 'cis/apm' and 'cis/api/' are considered to have the same base url, resulting in the request being ignored.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.