Kibana version:
N/A
Elasticsearch version:
7.17.7
APM Server version:
7.17.7
APM Agent language and version:
.net 1.19.0
Browser version:
N/A
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
nuget
Fresh install or upgraded from other version?
Fresh
Is there anything special in your setup?
running in k8s/arm64 (not related tho)
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I don't know if it's related to a recent .NET version (7.0.1) but APM server is spamming and dropping spans and it
and also spams ERROR level log: my Elasticsearch disks usage skyrocketed and made sentry drop all other exceptions due to transaction limit
THIS IS QUITE SERIOUS
Steps to reproduce:
- Use httpclient
- Wait till APM tries to ship the transaction
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):
[Error] {"PayloadSenderV2"} Failed sending event. Events intake API absolute URL: https://******.aws.elastic-cloud.com/intake/v2/events. APM Server response: status code: BadRequest, content:
"{"accepted":6,"errors":[{"message":"validation error: span: context: destination: service: 'resource' required","document":"{\"span\":{\"context\":{\"destination\":{\"service\":{\"name\":\"\",\"type\":\"\"}},\"service\":{\"target\":{}}},\"duration\":13.0439,\"id\":\"1cef3e702b0173c8\",\"name\":\"CallElasticsearch\",\"outcome\":\"unknown\",\"parent_id\":\"72b9660f6a9e5921\",\"links\":,\"sample_rate\":0.1,\"stacktrace\":[{\"abs_path\":\"\",\"classname\":\"Elastic.Apm.Model.Span\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"End\",\"lineno\":0,\"module\":\"Elastic.Apm, Version=1.19.0.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22\"},{\"abs_path\":\"\",\"classname\":\"Elastic.Apm.OpenTelemetry.ElasticActivityListener\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"\u003cget_ActivityStopped\u003eb__9_0\",\"lineno\":0,\"module\":\"Elastic.Apm, Version=1.19.0.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22\"},{\"abs_path\":\"\",\"classname\":\"System.Diagnostics.SynchronizedList1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"EnumWithAction\\",\\"lineno\\":0,\\"module\\":\\"System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Diagnostics.Activity\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"Stop\\",\\"lineno\\":0,\\"module\\":\\"System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"Elasticsearch.Net.Diagnostics.Diagnostic2\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"Dispose\",\"lineno\":0,\"module\":\"Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d\"},{\"abs_path\":\"\",\"classname\":\"Elasticsearch.Net.RequestPipeline+\u003cCallElasticsearchAsync\u003ed__671\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"CallElasticsearchAsync\\",\\"lineno\\":0,\\"module\\":\\"Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SetExistingTaskResult\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"Elasticsearch.Net.HttpConnection+\u003cRequestAsync\u003ed__151\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RequestAsync\\",\\"lineno\\":0,\\"module\\":\\"Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Net.Http.HttpClient+\u003c\u003cSendAsync\u003eg__Core|83_0\u003ed\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"\u003cSendAsync\u003eg__Core|83_0\\",\\"lineno\\":0,\\"module\\":\\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SetExistingTaskResult\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Http.RedirectHandler+\u003cSendAsync\u003ed__4\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SendAsync\",\"lineno\":0,\"module\":\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"MoveNext\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.AwaitTaskContinuation\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunOrScheduleAction\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.Task\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunContinuations\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Http.DiagnosticsHandler+\u003cSendAsyncCore\u003ed__10\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SendAsyncCore\",\"lineno\":0,\"module\":\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"MoveNext\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.AwaitTaskContinuation\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunOrScheduleAction\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.Task\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunContinuations\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.Task1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"TrySetResult\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Net.Http.HttpConnectionPool+\u003cSendWithVersionDetectionAndRetryAsync\u003ed__84\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"SendWithVersionDetectionAndRetryAsync\\",\\"lineno\\":0,\\"module\\":\\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SetExistingTaskResult\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Http.HttpConnection+\u003cSendAsyncCore\u003ed__61\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SendAsyncCore\",\"lineno\":0,\"module\":\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"MoveNext\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Security.SslStream+\u003cReadAsyncInternal\u003ed__1591\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"ReadAsyncInternal\\",\\"lineno\\":0,\\"module\\":\\"System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Net.Security.SslStream+\u003cEnsureFullTlsFrameAsync\u003ed__1571\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"EnsureFullTlsFrameAsync\",\"lineno\":0,\"module\":\"System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"InvokeContinuation\",\"lineno\":0,\"module\":\"System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"OnCompleted\",\"lineno\":0,\"module\":\"System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Sockets.SocketAsyncEngine\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"System.Threading.IThreadPoolWorkItem.Execute\",\"lineno\":0,\"module\":\"System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.ThreadPoolWorkQueue\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"Dispatch\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.PortableThreadPool+WorkerThread\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"WorkerThreadStart\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"}],\"subtype\":\"internal\",\"timestamp\":1671086277459371,\"trace_id\":\"e894041ca3d3deb51f8cca27845a3a08\",\"transaction_id\":\"72b9660f6a9e5921\",\"type\":\"app\",\"otel\":{\"spanKind\":\"Internal\"}}}"},{"message":"validation error: span: context: destination: service: 'resource' required","document":"{\"span\":{\"context\":{\"destination\":{\"service\":{\"name\":\"\",\"type\":\"\"}},\"service\":{\"target\":{}}},\"duration\":0.6126,\"id\":\"99d8fe70061d4278\",\"name\":\"ReceiveBody\",\"outcome\":\"unknown\",\"parent_id\":\"13fcbed6a472fe9a\",\"links\":,\"sample_rate\":0.1,\"subtype\":\"internal\",\"timestamp\":1671086277471750,\"trace_id\":\"e894041ca3d3deb51f8cca27845a3a08\",\"transaction_id\":\"72b9660f6a9e5921\",\"type\":\"app\",\"otel\":{\"spanKind\":\"Internal\"}}}"},{"message":"validation error: span: context: destination: service: 'resource' required","document":"{\"span\":{\"context\":{\"destination\":{\"service\":{\"name\":\"\",\"type\":\"\"}},\"service\":{\"target\":{}}},\"duration\":0.3352,\"id\":\"78c1fe1124f18c18\",\"name\":\"Deserialize\",\"outcome\":\"unknown\",\"parent_id\":\"1edebaee2578ae49\",\"links\":,\"sample_rate\":0.1,\"subtype\":\"internal\",\"timestamp\":1671086277471931,\"trace_id\":\"e894041ca3d3deb51f8cca27845a3a08\",\"transaction_id\":\"72b9660f6a9e5921\",\"type\":\"app\",\"otel\":{\"spanKind\":\"Internal\"}}}"},{"message":"validation error: span: context: destination: service: 'resource' required","document":"{\"span\":{\"context\":{\"destination\":{\"service\":{\"name\":\"\",\"type\":\"\"}},\"service\":{\"target\":{}}},\"duration\":11.406,\"id\":\"f8c5a06fd291b3e8\",\"name\":\"SendAndReceiveHeaders\",\"outcome\":\"unknown\",\"parent_id\":\"13fcbed6a472fe9a\",\"links\":,\"sample_rate\":0.1,\"stacktrace\":[{\"abs_path\":\"\",\"classname\":\"Elastic.Apm.Model.Span\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"End\",\"lineno\":0,\"module\":\"Elastic.Apm, Version=1.19.0.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22\"},{\"abs_path\":\"\",\"classname\":\"Elastic.Apm.OpenTelemetry.ElasticActivityListener\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"\u003cget_ActivityStopped\u003eb__9_0\",\"lineno\":0,\"module\":\"Elastic.Apm, Version=1.19.0.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22\"},{\"abs_path\":\"\",\"classname\":\"System.Diagnostics.SynchronizedList1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"EnumWithAction\\",\\"lineno\\":0,\\"module\\":\\"System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Diagnostics.Activity\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"Stop\\",\\"lineno\\":0,\\"module\\":\\"System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"Elasticsearch.Net.Diagnostics.Diagnostic2\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"Dispose\",\"lineno\":0,\"module\":\"Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d\"},{\"abs_path\":\"\",\"classname\":\"Elasticsearch.Net.HttpConnection+\u003cRequestAsync\u003ed__151\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RequestAsync\\",\\"lineno\\":0,\\"module\\":\\"Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Net.Http.HttpClient+\u003c\u003cSendAsync\u003eg__Core|83_0\u003ed\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"\u003cSendAsync\u003eg__Core|83_0\\",\\"lineno\\":0,\\"module\\":\\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SetExistingTaskResult\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Http.RedirectHandler+\u003cSendAsync\u003ed__4\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SendAsync\",\"lineno\":0,\"module\":\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"MoveNext\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.AwaitTaskContinuation\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunOrScheduleAction\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.Task\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunContinuations\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Http.DiagnosticsHandler+\u003cSendAsyncCore\u003ed__10\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SendAsyncCore\",\"lineno\":0,\"module\":\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"MoveNext\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.AwaitTaskContinuation\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunOrScheduleAction\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.Task\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"RunContinuations\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.Tasks.Task1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"TrySetResult\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Net.Http.HttpConnectionPool+\u003cSendWithVersionDetectionAndRetryAsync\u003ed__84\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"SendWithVersionDetectionAndRetryAsync\\",\\"lineno\\":0,\\"module\\":\\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"MoveNext\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.AwaitTaskContinuation\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunOrScheduleAction\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Threading.Tasks.Task\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"RunContinuations\\",\\"lineno\\":0,\\"module\\":\\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SetExistingTaskResult\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Http.HttpConnection+\u003cSendAsyncCore\u003ed__61\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"SendAsyncCore\",\"lineno\":0,\"module\":\"System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"MoveNext\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Security.SslStream+\u003cReadAsyncInternal\u003ed__1591\\",\\"filename\\":\\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\\",\\"function\\":\\"ReadAsyncInternal\\",\\"lineno\\":0,\\"module\\":\\"System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\\"},{\\"abs_path\\":\\"\\",\\"classname\\":\\"System.Net.Security.SslStream+\u003cEnsureFullTlsFrameAsync\u003ed__1571\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"EnsureFullTlsFrameAsync\",\"lineno\":0,\"module\":\"System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"InvokeContinuation\",\"lineno\":0,\"module\":\"System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"OnCompleted\",\"lineno\":0,\"module\":\"System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Net.Sockets.SocketAsyncEngine\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"System.Threading.IThreadPoolWorkItem.Execute\",\"lineno\":0,\"module\":\"System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.ThreadPoolWorkQueue\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"Dispatch\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"},{\"abs_path\":\"\",\"classname\":\"System.Threading.PortableThreadPool+WorkerThread\",\"filename\":\"/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.1/System.Private.CoreLib.dll\",\"function\":\"WorkerThreadStart\",\"lineno\":0,\"module\":\"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e\"}],\"subtype\":\"internal\",\"timestamp\":1671086277459817,\"trace_id\":\"e894041ca3d3deb51f8cca27845a3a08\",\"transaction_id\":\"72b9660f6a9e5921\",\"type\":\"app\",\"otel\":{\"spanKind\":\"Internal\"}}}"}]}
"