Hi there,
Kibana version: v 8.15.2
Elasticsearch version: v 8.15.2
APM Server version: v 8.15.2
APM Agent language and version: .net 8 version 1.31.0
Fresh install (POC)
We would like to introduce real-time observability in Elastic for our .NET Core 8 applications. To achieve this, we set up Elastic APM and used Elastic NuGet packages to instrument our app.
We performed two different setups, and in both cases, we were able to fully observe metrics and traces in Elastic (Kibana). Additionally, we created an agent configuration within Fleet, where we successfully assigned most of the settings (Recording, Transaction Sample Rate, Ignore Transactions Based on URLs, etc.). However, we encountered an issue when trying to configure the Stack Trace Limit and Span Stack Trace Minimum Duration settings.
We tried configuring the APM agent using both methods: Fleet Agent configuration and appsettings.json configuration.
We attempted to disable the Stack Trace Limit ("StackTraceLimit": "0") and experimented with other values, but the changes were not reflected in APM. Traces were still present, and for the same 1-hour load test (enabled or disabled with a 50ms value), we observed approximately the same results in the APM Storage Explorer.
We also tried optimizing Span Stack Trace Minimum Duration ("SpanStackTraceMinDuration": "-1") as described in the documentation. We set it to 5000ms and 5m, but the changes were not reflected in APM. Spans within transactions of less than 5ms were still present, and for the same 1-hour load test (with both the default value and -1ms value), we observed approximately the same results in the APM Storage Explorer.
StackTraceLimit (performance)
Setting this to 0 disables stack trace collection. Any positive integer value will be used as the maximum number of frames to collect. Setting it to -1 means that all frames will be collected.
SpanStackTraceMinDuration (performance)
If you would like to disable stack trace capturing only for spans, but still capture stack traces for errors, set the SpanStackTraceMinDuration (performance) config to -1.
Trace example for custom transaction:
Trace example for automatic transaction: The same result
What we would like to achieve:
We would like to achieve performance improvement in a way it is described in documentation, to be able to limit capture trace only for errors and on demand enable / disable it via Fleet agent configuration. At the moment, those two configurations do not reflect on trace results.
Scenario 1:
Nuget:
"Elastic.Apm.NetCoreAll"
"Elastic.Apm.StackExchange.Redis"
public static void ConfigureOpenTelemetryElastic(this WebApplicationBuilder builder)
{
builder.Services.AddAllElasticApm();
}
public static void UseOpenTelemetryElastic(this WebApplication app)
{
app.Services.GetService<IConnectionMultiplexer>().UseElasticApm();
}
Configuration:
"ElasticApm": {
"ServiceName": "AppName",
"SecretToken": Token",
"ServerUrl": "URL",
"Environment": "ENV",
"FlushInterval": "5000ms",
"MaxBatchEventCount": 100,
"MaxQueueEventCount": 1000,
"TraceContinuationStrategy": "restart_external",
"TransactionIgnoreUrls": "/swagger*, /_*, /health",
"ExcludedNamespaces": "System., Microsoft., MS., FSharp., Newtonsoft.Json, Serilog, NLog, Giraffe.",
"SpanStackTraceMinDuration": "-1",
"StackTraceLimit": "0"
}
Scenario 2:
Nuget:
"Elastic.Apm"
"Elastic.Apm.AspNetCore"
"Elastic.Apm.Elasticsearch"
"Elastic.Apm.EntityFrameworkCore"
"Elastic.Apm.Extensions.Hosting"
"Elastic.Apm.StackExchange.Redis"
Instrumentation:
public static void ConfigureOpenTelemetryElastic(this WebApplicationBuilder builder)
{
builder.Services.AddElasticApm(
new Elastic.Apm.DiagnosticSource.HttpDiagnosticsSubscriber(),
new Elastic.Apm.AspNetCore.DiagnosticListener.AspNetCoreDiagnosticSubscriber(),
new Elastic.Apm.EntityFrameworkCore.EfCoreDiagnosticsSubscriber(),
new Elastic.Apm.Instrumentations.SqlClient.SqlClientDiagnosticSubscriber(),
new Elastic.Apm.Elasticsearch.ElasticsearchDiagnosticsSubscriber()
);
}
public static void UseOpenTelemetryElastic(this WebApplication app)
{
app.Services.GetService<IConnectionMultiplexer>()?.UseElasticApm();
}
Configuration:
"ElasticApm": {
"ServiceName": "AppName",
"SecretToken": Token",
"ServerUrl": "URL",
"Environment": "ENV",
"FlushInterval": "5000ms",
"MaxBatchEventCount": 100,
"MaxQueueEventCount": 1000,
"TraceContinuationStrategy": "restart_external",
"TransactionIgnoreUrls": "/swagger*, /_*, /health",
"ExcludedNamespaces": "System., Microsoft., MS., FSharp., Newtonsoft.Json, Serilog, NLog, Giraffe.",
"SpanStackTraceMinDuration": "-1",
"StackTraceLimit": "0"
}