DB Span Stacktrace reporting wrong filename

[AbhijithCV]

Kibana version: 8.1.2
Elasticsearch version: 8.1.2
APM Server version: 8.1.2
APM Agent language and version: dotnet agent version: 1.18.0

I was doing profiler auto instrumentation of a netcoreapp3.1 application :
CourseLibraryAPI

While going through the db span stacktrace I found that filepath was wrong, even though the classname and function names are correct.

{
                "exclude_from_grouping" : false,
                "filename" : """C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.32\System.Private.CoreLib.dll""",
                "classname" : "CourseLibrary.API.Services.CourseLibraryRepository",
                "line" : {
                  "number" : 0
                },
                "function" : "GetCourses",
                "module" : "CourseLibrary.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
}

I am new to dotnet developement, can anyone explain why the filename is a dll not the .cs source code containing the class and the function - link to source code with the function

Environment variables used for profiler auto instrumentation:

set CORECLR_ENABLE_PROFILING=1
set CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
set CORECLR_PROFILER_PATH=D:\DotNet\elastic_apm_profiler_1.18.0\elastic_apm_profiler.dll
set ELASTIC_APM_PROFILER_HOME=D:\DotNet\elastic_apm_profiler_1.18.0
set ELASTIC_APM_PROFILER_INTEGRATIONS=D:\DotNet\elastic_apm_profiler_1.18.0\integrations.yml
set ELASTIC_APM_SPAN_STACK_TRACE_MIN_DURATION=0ms
set ELASTIC_APM_STACK_TRACE_LIMIT=-1
set ELASTIC_APM_SERVICE_NAME=CourseLibraryApp

[GregKalapos]

Hi @AbhijithCV,

this is because C# is a compiled language. Once you compile a .cs file, the output will be a .dll containing the byte-code - this is also known as the .NET assembly. So when your program runs, there is no .cs file with source code visible to the runtime since the runtime works with the compiled assemblies which are the .dll files.

The unfortunate consequence of this from an APM Agent's point of view is that the Elastic APM Agent is unable to determine from which .cs file a given method is coming from and only reports the .dll file.

Nevertheless, we still report the fully qualified class name in classname, so that should be enough to identify which class/method you see in a given stack frame.

So, overall this is expected behaviour.

[AbhijithCV]

@GregKalapos thank you for the reply.
I have also done profiler auto instrumentation for another asp .net core application targeting net6.0.
NopCommerce
For this application I was getting correct filename in the span stacktrace.

{
    "exclude_from_grouping": false,
    "abs_path": "D:\\DotNet\\nopCommerce_4.50.4_Source\\Libraries\\Nop.Services\\Localization\\LocalizationService.cs",
    "filename": "D:\\DotNet\\nopCommerce_4.50.4_Source\\Libraries\\Nop.Services\\Localization\\LocalizationService.cs",
    "classname": "Nop.Services.Localization.LocalizationService",
    "line": {
        "number": 116
    },
    "function": "ResourceValuesToDictionary",
    "module": "Nop.Services, Version=4.5.0.0, Culture=neutral, PublicKeyToken=null"
}

Can you explain this behaviour?
How does the agent determine to which source file the function belongs to?

[GregKalapos]

I have also done profiler auto instrumentation for another asp .net core application targeting net6.0 .
NopCommerce
For this application I was getting correct filename in the span stacktrace.
Can you explain this behaviour?

So, in additional to what I wrote in the previous answer, the agent actually tries to find the .cs file if possible - so indeed, the agent has a "best effort" mechanism to figure out the original .cs file, if this fails, the agent just reports the assembly (which is the .dll file) that is executed during runtime.

For determining the .cs file the agent uses this method: StackFrame.GetFileName Method (System.Diagnostics) | Microsoft Learn

That can determine the .cs file when debugging symbols are present and currently set up. If that method returns null, the agent just reports .dll files as the file name.

You can see this logic here: https://github.com/elastic/apm-agent-dotnet/blob/main/src/Elastic.Apm/Helpers/StacktraceHelper.cs#L87

How does the agent determine to which source file the function belongs to?

You opened another entry as well, I just see: Custom Stacktrace for Transaction

Is this question related to that entry. That indeed seems to be a bug (not sure yet, I need to dig deeper).

[AbhijithCV]

Is this question related to that entry. That indeed seems to be a bug (not sure yet, I need to dig deeper).

Yes its related.
I am provding steps to repoduce

Kibana version : 8.1.2
Elasticsearch version : 8.1.2
APM Server version : 8.1.2
APM Agent language and version : dotnet agent version: 1.18.0

1. Clone the repo - BuildingRESTfulAPIAspNetCore3/Finished sample/CourseLibrary at master · KevinDockx/BuildingRESTfulAPIAspNetCore3 · GitHub
2. Build the application - dotnet build
3. Configure elastic dotnet apm agent v1.18.0 for profiler auto instrumentatiion with following environment variables

set CORECLR_ENABLE_PROFILING=1
set CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
set CORECLR_PROFILER_PATH=<agent_path>\elastic_apm_profiler.dll
set ELASTIC_APM_PROFILER_HOME=<agent_path>
set ELASTIC_APM_PROFILER_INTEGRATIONS=<agent_path>\integrations.yml
set ELASTIC_APM_SPAN_STACK_TRACE_MIN_DURATION=0ms
set ELASTIC_APM_STACK_TRACE_LIMIT=-1
set ELASTIC_APM_SERVICE_NAME=CourseLibraryApp

4. Launch the dotnet application from directory CourseLibrary\CourseLibrary.API\bin\Debug\netcoreapp3.1 by using the command dotnet CourseLibrary.API.dll
5. Hit one of the rest api : http://localhost:5000/api/authors
6. Check the db span stacktrace from Kibana.

{
                "exclude_from_grouping" : false,
                "filename" : """C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.32\System.Private.CoreLib.dll""",
                "classname" : "CourseLibrary.API.Controllers.AuthorsController",
                "line" : {
                  "number" : 0
                },
                "function" : "GetAuthors",
                "module" : "CourseLibrary.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
              }

GetAuthors function is part of the application, but its filename is wrong.
Course Library application - GetAuthors function

Question: Agent should have resolved the filename right? since the symbol file .pdb is available.

[AbhijithCV]

Hi @GregKalapos,
Was this issue reproducable?