Kibana version: 7.17..0
Elasticsearch version: 7.17.0
APM Server version: 7.17.15
APM Agent language and version: .Net Core 6.0
Browser version: Chrome 119.0.6045.160
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Download page
Fresh install or upgraded from other version? Fresh
Is there anything special in your setup? Using reverse proxies in front of Elastic/Kibana/APM
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have two ASP .Net Core applications, both running .Net Core APM Agents:
- CIS
- CIS API
When running the applications on my local machine in IIS Express, APM entries are being delivered to Kibana, correlated as expected, and the Traces show the complete history including entries from both CIS and CIS API. Here's a specific instance of this for a particular request originating in CIS:
Trace:
This trace consists of 4 items:
A) Initial web request to CIS
B) Outgoing HTTP request from CIS to CIS API
C) Incoming request to CIS API
D) Database operation performed by the API
Once deploying the same code to the server (running IIS), however, a few things happen:
-
The Trace only shows a single entry from CIS ('A' from above)
-
Querying the Logs for records with the same trace.id as in the above record, however:
a. There are entry records being captured for the CIS API incoming request and database operation (records 'C' and 'D' from above)
b. No event record exists for the outgoing HTTP request event from CIS to the CIS API (record 'B' from above)
My questions are:
- Why are my Traces showing as incomplete in Kibana, despite having correlated entries with the same trace.id?
- Why are not all outgoing HTTP requests from CIS being captured?
Things I've tried so far:
- The same missing trace entry behavior also occurs if I run CIS locally, hitting the deployed version of CIS API.
- In the deployed instances, CIS does indeed capture a few outgoing HTTP requests (to other APM tracked services), but the majority of them are not captured (including all to the API).
- Examining requests from a Locally running CIS to both the Local and Deployed versions of the API, there are inconsistent inclusions of the 'elastic-apm-traceparent' request header. The request to the Local API includes an 'elastic-apm-traceparent' header, however the request to the Deployed API does not.
Steps to reproduce:
- Both applications are configured using
app.UseAllElasticApm(configuration);
- Calls to the API are made using using the following code:
public async Task<T> GetStats<T>(string query_params)
{
var endpoint = $"Stats?{query_params}";
var request = new HttpRequestMessage(HttpMethod.Get, endpoint);
var client = _clientFactory.CreateClient(HttpClients.CISApi);
var response = await client.SendAsync(request);
var json = await response.Content.ReadAsStringAsync();
var data = JsonConvert.DeserializeObject<T>(json, new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Ignore
});
return data;
}
Errors in browser console (if relevant): N/A
Provide logs and/or server output (if relevant): N/A
Happy to provide more info, if needed. Thanks!