APM Agent language and version: Elastic APM .Net 1.28.0
Original install method and version: Nuget package
Elastic.Apm 1.28.0
Elsatic.Apm.AspNetFullFramework 1.28.0
We are using Elastic.Apm.AspNetFullFramework in our asp.net web api targeting .net 4.8. We initialize the agent using web.config method via ElasticApmModule:
<modules>
<add name="ElasticApmModule" type="Elastic.Apm.AspNetFullFramework.ElasticApmModule, Elastic.Apm.AspNetFullFramework" />
</modules>
Issue: After enabling APM we noticed memory leak issue similar to #850 but may not be related to spans. Analysing the memory dump of w3wp.exe process we can see EF related entity objects are retained and using WinDBG we could see its tracing back to Elastic APM namespace along with some timer functions and i have provided windbg output for one of the object for reference at the bottom.
While we troubleshoot the issue we want to disable SqlEventListener temporarily and noticed there's currently no config to do so. Do anyone suggest any alternative to disable it and also provide some guidance on the memory leak.
WinDdg output of GC root command for one of the retained object:
0:000> !gcroot 0x0000024890fc1968
Thread 2aac:
000000c1f2fbeb30 00007ff92855c90f System.Net.TimerThread.ThreadProc()
rdi: (interior)
-> 0000024a90bd8ff0 System.Object[]
-> 0000024990e56818 System.PinnableBufferCacheEventSource
-> 0000024790f756a0 System.Diagnostics.Tracing.EventDispatcher
-> 0000024790f75220 Elastic.Apm.Instrumentations.SqlClient.SqlEventListener
-> 0000024890d6e958 Elastic.Apm.ApmAgent
-> 0000024990eb25d0 Elastic.Apm.AspNetFullFramework.AgentComponentsUsingHttpContext
-> 0000024790f6f080 Elastic.Apm.BackendComm.CentralConfig.CentralConfigurationFetcher
-> 0000024790f6f688 System.Threading.CancellationTokenSource
-> 0000024690df3a48 System.Threading.SparselyPopulatedArray`1[[System.Threading.CancellationCallbackInfo, mscorlib]][]
-> 0000024690df3a80 System.Threading.SparselyPopulatedArray`1[[System.Threading.CancellationCallbackInfo, mscorlib]]
-> 0000024690df3aa0 System.Threading.SparselyPopulatedArrayFragment`1[[System.Threading.CancellationCallbackInfo, mscorlib]]
-> 0000024690df3ad0 System.Threading.CancellationCallbackInfo[]
-> 00000246a4c733f8 System.Threading.CancellationCallbackInfo
-> 00000246a4c73380 System.Threading.Tasks.Task+DelayPromise
-> 00000246a4c73430 System.Threading.Timer
-> 00000246a4c734a8 System.Threading.TimerHolder
-> 00000246a4c73450 System.Threading.TimerQueueTimer
-> 0000024690de3328 System.Threading.TimerQueueTimer
-> 00000246a3f979f0 System.Threading.TimerQueueTimer
-> 0000024991a1a368 System.Threading.TimerQueueTimer
-> 0000024991a19270 System.Threading.TimerQueueTimer
-> 0000024991a17cd8 System.Threading.TimerQueueTimer
-> 0000024991a16740 System.Threading.TimerQueueTimer
-> 0000024991a151a8 System.Threading.TimerQueueTimer
-> 0000024991a15148 System.Threading.TimerCallback
-> 0000024991a13ec0 System.Runtime.Caching.CacheExpires
-> 0000024990e91208 System.Runtime.Caching.MemoryCacheStore
-> 0000024990e91270 System.Collections.Hashtable
-> 00000246972f0a20 System.Collections.Hashtable+bucket[]
-> 00000246a3f3d5d8 System.Runtime.Caching.MemoryCacheEntry
-> 00000246a3f3d548 System.Collections.Generic.List`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000247a4cf9b28 Test.Domain.SomeDomainTable[]
-> 00000247a5389388 <Unloaded Type>
-> 00000247a4c2ee38 <Unloaded Type>
-> 00000247a4c5fbf8 <error>
-> 00000247a4c5fc50 System.Data.Entity.Core.Objects.DataClasses.RelationshipManager
-> 00000247a4c601c8 System.Collections.Generic.List`1[[System.Data.Entity.Core.Objects.DataClasses.RelatedEnd, EntityFramework]]
-> 00000247a4c601f0 System.Data.Entity.Core.Objects.DataClasses.RelatedEnd[]
-> 00000247a4c60128 System.Data.Entity.Core.Objects.DataClasses.EntityCollection`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000246a401fe20 System.Data.Entity.Core.Objects.ObjectContext
-> 0000024890fc1968 System.Data.Entity.Core.Metadata.Edm.MetadataWorkspace
HandleTable:
00000246906a3790 (strong handle)
-> 0000024991a153e8 System.Runtime.Caching.MemoryCacheStore
-> 0000024991a15450 System.Collections.Hashtable
-> 0000024794e5a6c0 System.Collections.Hashtable+bucket[]
-> 00000246a33cb9f8 System.Runtime.Caching.MemoryCacheEntry
-> 00000248a429a900 System.Collections.Generic.List`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000248a45b8a98 Test.Domain.SomeDomainTable[]
-> 00000248a43411d0 <Unloaded Type>
-> 00000248a43475a8 <Unloaded Type>
-> 00000248a4356868 <error>
-> 00000248a43568c0 System.Data.Entity.Core.Objects.DataClasses.RelationshipManager
-> 00000248a4356e38 System.Collections.Generic.List`1[[System.Data.Entity.Core.Objects.DataClasses.RelatedEnd, EntityFramework]]
-> 00000248a4356e60 System.Data.Entity.Core.Objects.DataClasses.RelatedEnd[]
-> 00000248a4356d98 System.Data.Entity.Core.Objects.DataClasses.EntityCollection`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000248a486ccd0 System.Data.Entity.Core.Objects.ObjectContext
-> 0000024890fc1968 System.Data.Entity.Core.Metadata.Edm.MetadataWorkspace
00000246906c15d8 (pinned handle)
-> 0000024ab0be7778 System.Object[]
-> 0000024690d247a0 System.Lazy`1[[System.Data.Entity.Infrastructure.Interception.DbDispatchers, EntityFramework]]
-> 0000024690d24be0 System.Lazy`1+Boxed[[System.Data.Entity.Infrastructure.Interception.DbDispatchers, EntityFramework]]
-> 0000024690d24820 System.Data.Entity.Infrastructure.Interception.DbDispatchers
-> 0000024690d248f8 System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher
-> 0000024690d24910 System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1[[System.Data.Entity.Infrastructure.Interception.IDbCommandInterceptor, EntityFramework]]
-> 00000249a862fda0 System.Collections.Generic.List`1[[System.Data.Entity.Infrastructure.Interception.IDbCommandInterceptor, EntityFramework]]
-> 0000024ab17e1c20 System.Data.Entity.Infrastructure.Interception.IDbCommandInterceptor[]
-> 00000247915f2d68 System.Data.Entity.Infrastructure.Interception.DatabaseLogFormatter
-> 00000247915f2d00 System.Action`1[[System.String, mscorlib]]
-> 00000247915f2cd0 Framework.Data.Repositories.Repository`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000247915ed980 Test.Domain.Test_Entities
-> 00000247915ede70 System.Data.Entity.Internal.LazyInternalContext
-> 00000247915edf50 System.Collections.Generic.Dictionary`2[[System.Type, mscorlib],[System.Data.Entity.Internal.Linq.IInternalSetAdapter, EntityFramework]]
-> 00000247915f0b30 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[System.Data.Entity.Internal.Linq.IInternalSetAdapter, EntityFramework]][]
-> 00000247915f23b8 System.Data.Entity.DbSet`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000247915f2370 System.Data.Entity.Internal.Linq.InternalSet`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000247915f3ad0 System.Data.Entity.Core.Objects.ObjectQuery`1[[Test.Domain.SomeDomainTable, Test.Domain]]
-> 00000247915f3b00 System.Data.Entity.Core.Objects.EntitySqlQueryState
-> 00000247915f3498 System.Data.Entity.Core.Objects.ObjectContext
-> 0000024890fc1968 System.Data.Entity.Core.Metadata.Edm.MetadataWorkspace
Found 3 unique roots (run '!GCRoot -all' to see all roots).