We want to trace all of our applications (built with OutSystems lowcode platform). So we used the auto instrumentation profiler for IIS 10.0. We are able to receive metrics, but traces are missing. The logs showed us that it is loading/tracing 600+ modules and we saw sql executescaler traces in the logs. So we believe it is able to hook into our applications.
After adding the apm listener to the web.config of one of our applications we noticed that threads are being aborted on reading centralconfig and I assume the API function for sending the trace to Elastic.
Kibana version: 7.16.3
Elasticsearch version: 7.16.3
APM Server version: 7.16.2
APM Agent language and version: 1.13.0 CLR profiler
Original install method (e.g. download page, yum, deb, from source, etc.) and version: from source (github apm agent)
Fresh install or upgraded from other version? Fresh
Is there anything special in your setup? No. Just IIS10
Runtime name & version: .NET Framework 4.8.4420.0
Framework name: ASP.NET
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): We are only receiving metrics from the agent. No apm traces. After adding the tracelistener to the web.config we saw these errors occuring:
[2022-01-27 09:41:35.706 +01:00][Error] - {ExceptionUtils.DoSwallowingExceptions} `ElasticApmCentralConfigurationFetcher' (managed ID: 68) thread entry method is about to exit because of exception+-> Exception: System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken)
at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Collections.Concurrent.BlockingCollection`1.TryTakeWithNoTimeValidation(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken, CancellationTokenSource combinedTokenSource)
at System.Collections.Concurrent.BlockingCollection`1.<GetConsumingEnumerable>d__68.MoveNext()
at Elastic.Apm.Helpers.SingleThreadTaskScheduler.<ThreadMain>b__14_0() in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1594\apm-agent-dotnet\src\Elastic.Apm\Helpers\SingleThreadTaskScheduler.cs:line 50
at Elastic.Apm.Helpers.ExceptionUtils.DoSwallowingExceptions(IApmLogger loggerArg, Action action, Boolean shouldSwallowCancellation, String dbgCallerMethodName) in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1594\apm-agent-dotnet\src\Elastic.Apm\Helpers\ExceptionUtils.cs:line 29
[2022-01-27 09:41:35.706 +01:00][Error] - {ExceptionUtils.DoSwallowingExceptions} `ElasticApmPayloadSenderV2' (managed ID: 67) thread entry method is about to exit because of exception+-> Exception: System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken)
at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Collections.Concurrent.BlockingCollection`1.TryTakeWithNoTimeValidation(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken, CancellationTokenSource combinedTokenSource)
at System.Collections.Concurrent.BlockingCollection`1.<GetConsumingEnumerable>d__68.MoveNext()
at Elastic.Apm.Helpers.SingleThreadTaskScheduler.<ThreadMain>b__14_0() in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1594\apm-agent-dotnet\src\Elastic.Apm\Helpers\SingleThreadTaskScheduler.cs:line 50
at Elastic.Apm.Helpers.ExceptionUtils.DoSwallowingExceptions(IApmLogger loggerArg, Action action, Boolean shouldSwallowCancellation, String dbgCallerMethodName) in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1594\apm-agent-dotnet\src\Elastic.Apm\Helpers\ExceptionUtils.cs:line 29
Steps to reproduce:
- We used this configuration to install the profiler:
$appcmd = "$($env:systemroot)\system32\inetsrv\AppCmd.exe"
$appPool = "Apppoolname"
$profilerHomeDir = "E:\ElasticStack\elastic_apm_profiler"
$environment = @{
COR_ENABLE_PROFILING = "1"
COR_PROFILER_PATH = "$profilerHomeDir\elastic_apm_profiler.dll"
COR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
COMPlus_LoaderOptimization = "1"
ELASTIC_APM_PROFILER_HOME = "$profilerHomeDir\"
ELASTIC_APM_PROFILER_INTEGRATIONS = "$profilerHomeDir\integrations.yml"
ELASTIC_APM_PROFILER_LOG_DIR="$profilerHomeDir\logs"
ELASTIC_APM_PROFILER_LOG="trace"
ELASTIC_APM_PROFILER_LOG_IL="1"
}
$environment.Keys | ForEach-Object {
write-output "[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
#delete entry
& $appcmd set config -section:system.applicationHost/applicationPools /-"[name='$appPool'].environmentVariables.[name='$_']"
#add entry
& $appcmd set config -section:system.applicationHost/applicationPools /+"[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
}
& $appcmd list config -section:system.applicationHost/applicationPools/[name='$appPool']
# edit webconfig
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<add key="ElasticApm:ServerUrl" value="https://********.apm.westeurope.azure.elastic-cloud.com" />
<add key="ElasticApm:SecretToken" value="********" />
<add key="ElasticApm:Recording" value="true" />
<add key="ElasticApm:Enabled" value="true" />
<add key="ElasticApm:ServiceName" value="HyperionDevAPM" />
<add key="ElasticApm:HostName" value="hyperion-dev" />
<add key="ElasticApm:Environment" value="Hyperion Development" />
</appSettings>
</configuration>
Hope this is enough information to figure out the issue or what we did wrong. Please let me know if more information is required. Thanks in advance for any assistance!
