Apm-agent-dotnet Errors after updating from 1.14.1 to 1.15.0+

Elastic Observability APM
,
1

Hi, after updating one of our .NET 6 services to use Elastic.Apm.Azure.* v1.15.0 (from 1.14.1), we started seeing this in our logs (and do on v1.16.0 as well) about every 30 seconds:

{BackendCommComponentBase (CentralConfigurationFetcher)} WorkLoop Current thread: `ElasticApmCentralConfigurationFetcher' (managed ID: 13)
System.AggregateException: One or more errors occurred. (A task was canceled.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.Task.GetExceptions(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Elastic.Apm.BackendComm.BackendCommComponentBase.WorkLoop()
   at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Elastic.Apm.BackendComm.BackendCommComponentBase.WorkLoop()

Does this have any relation to https://github.com/elastic/apm-agent-dotnet/issues/1626, or is this a separate issue? We've downgraded the library to v1.14.1 in the meantime.

Thanks,
Greg

2

Hi @GregS1,

I think it's related, but not exactly the same - this is not the same log line as the one from the linked issue. Have you tried turning on central config?

Having this every 30 sec (that's when the agent fetches configs periodically) is clearly not ok. I'll look into this.

3

Unfortunately the relevant servers in question for this error are controlled by our cloud infrastructure group so I don't have an easy way to try adding a central config, and I'm not sure whether they would do so, though I could check.

When I try to reproduce this via a local Elastic stack running via Docker Compose, I get the error noted in your Github link, rather than the one noted in this thread.

4

Hey @GregS1

I looked into this more - I wasn't able to reproduce the problem locally, but the log with the exception details you pasted is already useful enough - thanks for that!

I opened a PR to improve this: Change logs around central config by gregkalapos · Pull Request #1732 · elastic/apm-agent-dotnet · GitHub

Two things will change:

  • The recurring error log is now change to debug (as it was before) - so this WorkLoop related log will only be visible on debug level (which you should only turn on for debugging).
  • As described in the original issue, when failing to fetch central config due to HttpRequestException (e.g. because central config isn't even enabled) the agent won't print an error. This is an expected scenario, so this will also only show up in debug logs.

Likely the 1. one will help in your scenario.

5

Sounds good! That should indeed take care of the issue; thank you. I'll keep an eye out for the next release :+1:

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.