Elastic APM IIS Universal Profiler for both .NET Core and .NET Framework

Hi,

I have a specific case that I'm not sure how to move forward.

In short, I'm trying to integrate Elastic APM to IIS as a universal profiler using Windows system environments. There're more than one applications running on .NET Core and .NET Framework on lots of Windows servers. The list is so long to edit all application pool environment variables manually.

I'm looking for a implementation which will provide IIS universal profiler for both .NET Core and .NET Framework pools.

Would adding both CORECLR_xxx_xxx and COR_xxx_xxx environment variables work?

Thank you.

Hi, @onurcan

To clarify, do you mean the auto-instrumentation, which uses the .NET runtime profiling APIs to inject our agent and instrumentation?

You can add both .NET Framework and .NET (Core) environment variables, which will then attach the profiler to any .NET app. Note that we recommend setting these per app pool in IIS rather than as system environment variables because the latter will attempt to attach to any .NET app on the server, including things like Powershell, which you likely want to avoid.

You could fairly easily script this in Powershell to iterate all app pools and attach the profiling environment variables.

Does that answer your question?

Hi, @stevejgordon

Thank you for your answer. Yes, I meant the auto-instrumentation and I got the answer from your reply.

It's okay for us to collect everything on the server for now. However, after adding global system variables for both .NET Core and .NET Framework applications auto profiler does not seem to collect anything from all application pools.

Let's say there're 10 different application pools that I'm sure there are lots of transactions on those but Elastic APM agent collects and sends only for 2-3 of them.

Also, I'm quite new to the Windows IIS environment so not sure how to iterate all app pools via Powershell script.

You're welcome, @onurcan.

Was IIS restarted after adding the system env vars?

Is there any commonality between the app pools that work and those that do not?

You should be able to find logs for the profiler in %PROGRAMDATA%\elastic\apm-agent-dotnet\logs, or you can configure a custom directory using ELASTIC_APM_PROFILER_LOG_DIR (see docs for more detail).

I suggest reviewing the logs to see if that identifies the cause of the problem.

Yes, IIS was restarted after adding the system env vars.

There is no application pool specific differences between the app pools that work and those don't.

I've been reviewing the profiler logs but in fact there are no logs for the app pools that do not work.

There's only one specific case which I can see the app pool on Kibana APM Services dashboard with no transactions and a created log file on the server.
When I cross check the log files using that specific app pool, I realized that there's a line for working app pools:

[Debug] - {DiagnosticInitializer} 'Microsoft.AspNetCore' subscribed by: AspNetCoreDiagnosticListener

For those which do not work, this line does not exist or logged.

If there are no logs for the failing app pools then the env vars might not be being picked up for some reason. Hard to say what that might be. If the profiler tries to attach, there should be a log. Are all the apps 32-bit/64-bit? The profiler is specific to the architecture, so you need to attach the right one. If the apps differ, that might be an issue with this approach. You could also check the Windows Event Log to see if that includes any errors relating to the profiler.

After some digging, I found that Event Viewer logged something like this:

.NET Runtime version 4.0.30319.0 - Loading profiler failed during CoCreateInstance.  Profiler CLSID: '{3eb5fcbe-edc7-40b6-b535-4d63c02246ed}'.  HRESULT: 0x8007007e.  Process ID (decimal): 2584.  Message ID: [0x2504].

If I don't get it wrong, as stated in auto instrumentation documentation, it does not support this .NET Runtime version 4.0.x.0.

Yes, it appears the server is running an unsupported .NET Framework. Their build numbers can be tricky, but this resolves .NET 4.0 based on what I could find. Which OS and OS build is this running on? We only support .NET 4.6.2 and newer in line with Microsoft's support policy of .NET Framework.

The HResult suggests the DLL could not be loaded, which might be due to the outdated runtime version. Another thing to verify is that the App Pool Identify has permissions to the directory where the profiler is unzipped. That could also prevent the DLL loading.

Hi @stevejgordon,

It makes sense that the application pools that do not work is running an unsupported .NET Framework or .NET Core.

In our scenario, it is unlikely that the App Pool Identity does not have the permissions to the directory where the profiler resides. We can get transactions from some of the app pools from the same server.

We'll be working with the engineering team to upgrade the .NET runtime versions, test the auto profiler and move forward.

Thank you for your time.

You're welcome, @onurcan