Application startup exception

Vincent7326 · December 20, 2022, 12:23pm

I just upgrade elasticsearch, kibana and elasticagent(fleet and apm) from 8.4.3 to 8.5.3
Then I cannot receive any apm transactions ...

After, I downgrade elastic agent to 8.4.3, and my application got problem ...
The log list below

Kibana version:
8.5.3

Elasticsearch version:
8.5.3

APM Server version:
8.4.3

APM Agent language and version:
.net core, 1.18.0/1.19.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?
upgrade from 8.4.3 to 8.5.3

Provide logs and/or server output (if relevant):

[DEBUG] 2022-12-20T08:04:05.3774906-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} Service name provided in configuration is MyProject
[INFORMATION] 2022-12-20T08:04:05.4924238-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The service version will be automatically discovered.
[INFORMATION] 2022-12-20T08:04:05.4965594-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The automatically discovered service version is 1.0.0
[DEBUG] 2022-12-20T08:04:05.5721500-04:00 [] - {SystemInfoHelper} Found Kubernetes pod UID: fabd28e4-5866-47c8-bda1-fa8a64499bf7
[DEBUG] 2022-12-20T08:04:05.6887434-04:00 [] - {BackendCommUtils} Building HTTP client with BaseAddress: http://apm-server/ for BackendCommComponentBase (PayloadSenderV2)...
[DEBUG] 2022-12-20T08:04:05.6973034-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} UseWindowsCredentials configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.8853329-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} GlobalLabels configuration option doesn't have a valid value - using default (empty map)
[DEBUG] 2022-12-20T08:04:05.8940048-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} MaxQueueEventCount configuration option doesn't have a valid value - using default: 1000
[DEBUG] 2022-12-20T08:04:05.8964154-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} MaxBatchEventCount configuration option doesn't have a valid value - using default: 10
[DEBUG] 2022-12-20T08:04:05.9011967-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} FlushInterval configuration option doesn't have a valid value - using default: 00:00:10
[DEBUG] 2022-12-20T08:04:05.9062037-04:00 [] - {PayloadSenderV2} Using the following configuration options: Events intake API absolute URL: http://apm-server/intake/v2/events, FlushInterval: 10s, MaxBatchEventCount: 10, MaxQueueEventCount: 1000
[DEBUG] 2022-12-20T08:04:05.9701147-04:00 [] - {BackendCommComponentBase (PayloadSenderV2)} Waiting for work loop started event...
[DEBUG] 2022-12-20T08:04:05.9717383-04:00 [] - {BackendCommComponentBase (PayloadSenderV2)} Signaling work loop started event...
[DEBUG] 2022-12-20T08:04:05.9724314-04:00 [] - {BackendCommComponentBase (PayloadSenderV2)} Work loop started signaled
[DEBUG] 2022-12-20T08:04:05.9860694-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9864938-04:00 [] - {BackendCommUtils} Building HTTP client with BaseAddress: http://apm-server/ for BackendCommComponentBase (CentralConfigurationFetcher)...
[DEBUG] 2022-12-20T08:04:05.9868954-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} UseWindowsCredentials configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9884932-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9887207-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9889403-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9893038-04:00 [] - {CentralConfigurationFetcher} Central configuration feature is enabled because CentralConfig option's value is True (default value is True)
[DEBUG] 2022-12-20T08:04:05.9895850-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9899577-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:06.0775425-04:00 [] - {CentralConfigurationFetcher} Combined absolute URL for APM Server get central configuration endpoint: `http://apm-server/config/v1/agents?service.name=MyProject&service.environment=Staging'. Service: Service{Name: MyProject, Version: 1.0.0, Environment: Staging, Runtime: Runtime{Name: .NET 5, Version: 5.0.17}, Framework: null, Agent: AgentC{Name: dotnet, Version: 1.18.0+31bb64353e9cc005906ea29bc086604fcdeda388}, Language: null, Node: Node{ConfiguredName: null}}.
[DEBUG] 2022-12-20T08:04:06.0794634-04:00 [] - {BackendCommComponentBase (CentralConfigurationFetcher)} Waiting for work loop started event...
[DEBUG] 2022-12-20T08:04:06.0796484-04:00 [] - {BackendCommComponentBase (CentralConfigurationFetcher)} Signaling work loop started event...
[DEBUG] 2022-12-20T08:04:06.0802032-04:00 [] - {BackendCommComponentBase (CentralConfigurationFetcher)} Work loop started signaled
[INFORMATION] 2022-12-20T08:04:07.1741186-04:00 [] - {MetricsCollector} Collecting metrics in 30000 milliseconds interval
[DEBUG] 2022-12-20T08:04:07.2879246-04:00 [] - {HttpDiagnosticListener} Current runtime is not detected as Full Framework - returning implementation for Core. RuntimeInformation.FrameworkDescription: {RuntimeInformation.FrameworkDescription}
[DEBUG] 2022-12-20T08:04:07.3770592-04:00 [] - {DiagnosticInitializer} Subscribed Elastic.Apm.DiagnosticListeners.HttpDiagnosticListenerCoreImpl to `HttpHandlerDiagnosticListener' events source
[DEBUG] 2022-12-20T08:04:07.3895688-04:00 [] - {DiagnosticInitializer} Subscribed Elastic.Apm.SqlClient.SqlClientDiagnosticListener to `SqlClientDiagnosticListener' events source
[DEBUG] 2022-12-20T08:04:07.4643053-04:00 [] - {DiagnosticInitializer} Subscribed Elastic.Apm.SqlClient.SqlClientDiagnosticListener to `SqlClientDiagnosticListener' events source
[FATAL] 2022-12-20T08:04:07.4880027-04:00 [] - Application startup exception
System.MissingFieldException: Field not found: 'Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1.ckingField'.
   at Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
   at Elasticsearch.Net.Diagnostics.AuditDiagnosticObserver..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
   at Elastic.Apm.Elasticsearch.AuditDiagnosticsListener..ctor(IApmAgent agent)
   at Elastic.Apm.Elasticsearch.ElasticsearchDiagnosticsSubscriber.Subscribe(IApmAgent agentComponents)
   at Elastic.Apm.ApmAgentExtensions.Subscribe(IApmAgent agent, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, ApmAgent agent, IApmLogger logger, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, IConfiguration configuration, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.NetCoreAll.ApmMiddlewareExtension.UseAllElasticApm(IApplicationBuilder builder, IConfiguration configuration)
   at MyProject.Backend.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, ICorsOriginValidator originValidator, ILogger`1 logger) in /src/MyProject.Backend/Startup.cs:line 287
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Unhandled exception. System.MissingFieldException: Field not found: 'Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1.ckingField'.
   at Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
   at Elasticsearch.Net.Diagnostics.AuditDiagnosticObserver..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
   at Elastic.Apm.Elasticsearch.AuditDiagnosticsListener..ctor(IApmAgent agent)
   at Elastic.Apm.Elasticsearch.ElasticsearchDiagnosticsSubscriber.Subscribe(IApmAgent agentComponents)
   at Elastic.Apm.ApmAgentExtensions.Subscribe(IApmAgent agent, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, ApmAgent agent, IApmLogger logger, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, IConfiguration configuration, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.NetCoreAll.ApmMiddlewareExtension.UseAllElasticApm(IApplicationBuilder builder, IConfiguration configuration)
   at MyProject.Backend.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, ICorsOriginValidator originValidator, ILogger`1 logger) in /src/MyProject.Backend/Startup.cs:line 287
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at MyProject.Backend.Program.Main(String[] args) in /src/MyProject.Backend/Program.cs:line 14

GregKalapos · December 20, 2022, 3:05pm

Hi @Vincent7326,

sorry to hear this - this should not happen, so we are looking into this. It's likely something we'll need to fix.

Could you tell me which .NET Elasticsearch client package you use including the version?

You'd see this in your application's .csproj file.

It should be something like Elasticsearch.Net , NEST, or Elastic.Clients.Elasticsearch. Which one of those you have in your app, and what's the version of those?

This would help us to reproduce the issue and then find a fix.

Thanks, and again sorry for the trouble.

Vincent7326 · December 21, 2022, 3:27am

I just install these two packages
Elastic.Apm.dll - 1.18.0.0
Elastic.Apm.NetCoreAll.dll - 1.18.0.0

After build I also have
Elasticsearch.Net.dll - 7.6.0.0
Don't have NEST.dll, Elastic.Clients.Elasticsearch.dll

I also tried Elastic.Apm/Elastic.Apm.NetCore version 1.19 and it didn't improve.

I am using UseElasticApm instead of UseElasticApmAll and not using AuditDiagnosticsListener for now as a temporary solution.

system · January 10, 2023, 11:27pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.