I just upgrade elasticsearch, kibana and elasticagent(fleet and apm) from 8.4.3 to 8.5.3
Then I cannot receive any apm transactions ...
After, I downgrade elastic agent to 8.4.3, and my application got problem ...
The log list below
Kibana version:
8.5.3
Elasticsearch version:
8.5.3
APM Server version:
8.4.3
APM Agent language and version:
.net core, 1.18.0/1.19.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Fresh install or upgraded from other version?
upgrade from 8.4.3 to 8.5.3
Provide logs and/or server output (if relevant):
[DEBUG] 2022-12-20T08:04:05.3774906-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} Service name provided in configuration is MyProject
[INFORMATION] 2022-12-20T08:04:05.4924238-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The service version will be automatically discovered.
[INFORMATION] 2022-12-20T08:04:05.4965594-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The automatically discovered service version is 1.0.0
[DEBUG] 2022-12-20T08:04:05.5721500-04:00 [] - {SystemInfoHelper} Found Kubernetes pod UID: fabd28e4-5866-47c8-bda1-fa8a64499bf7
[DEBUG] 2022-12-20T08:04:05.6887434-04:00 [] - {BackendCommUtils} Building HTTP client with BaseAddress: http://apm-server/ for BackendCommComponentBase (PayloadSenderV2)...
[DEBUG] 2022-12-20T08:04:05.6973034-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} UseWindowsCredentials configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.8853329-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} GlobalLabels configuration option doesn't have a valid value - using default (empty map)
[DEBUG] 2022-12-20T08:04:05.8940048-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} MaxQueueEventCount configuration option doesn't have a valid value - using default: 1000
[DEBUG] 2022-12-20T08:04:05.8964154-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} MaxBatchEventCount configuration option doesn't have a valid value - using default: 10
[DEBUG] 2022-12-20T08:04:05.9011967-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} FlushInterval configuration option doesn't have a valid value - using default: 00:00:10
[DEBUG] 2022-12-20T08:04:05.9062037-04:00 [] - {PayloadSenderV2} Using the following configuration options: Events intake API absolute URL: http://apm-server/intake/v2/events, FlushInterval: 10s, MaxBatchEventCount: 10, MaxQueueEventCount: 1000
[DEBUG] 2022-12-20T08:04:05.9701147-04:00 [] - {BackendCommComponentBase (PayloadSenderV2)} Waiting for work loop started event...
[DEBUG] 2022-12-20T08:04:05.9717383-04:00 [] - {BackendCommComponentBase (PayloadSenderV2)} Signaling work loop started event...
[DEBUG] 2022-12-20T08:04:05.9724314-04:00 [] - {BackendCommComponentBase (PayloadSenderV2)} Work loop started signaled
[DEBUG] 2022-12-20T08:04:05.9860694-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9864938-04:00 [] - {BackendCommUtils} Building HTTP client with BaseAddress: http://apm-server/ for BackendCommComponentBase (CentralConfigurationFetcher)...
[DEBUG] 2022-12-20T08:04:05.9868954-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} UseWindowsCredentials configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9884932-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9887207-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9889403-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9893038-04:00 [] - {CentralConfigurationFetcher} Central configuration feature is enabled because CentralConfig option's value is True (default value is True)
[DEBUG] 2022-12-20T08:04:05.9895850-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:05.9899577-04:00 [] - {AbstractConfigurationReader (MicrosoftExtensionsConfig)} CentralConfig configuration option doesn't have a valid value - using default: True
[DEBUG] 2022-12-20T08:04:06.0775425-04:00 [] - {CentralConfigurationFetcher} Combined absolute URL for APM Server get central configuration endpoint: `http://apm-server/config/v1/agents?service.name=MyProject&service.environment=Staging'. Service: Service{Name: MyProject, Version: 1.0.0, Environment: Staging, Runtime: Runtime{Name: .NET 5, Version: 5.0.17}, Framework: null, Agent: AgentC{Name: dotnet, Version: 1.18.0+31bb64353e9cc005906ea29bc086604fcdeda388}, Language: null, Node: Node{ConfiguredName: null}}.
[DEBUG] 2022-12-20T08:04:06.0794634-04:00 [] - {BackendCommComponentBase (CentralConfigurationFetcher)} Waiting for work loop started event...
[DEBUG] 2022-12-20T08:04:06.0796484-04:00 [] - {BackendCommComponentBase (CentralConfigurationFetcher)} Signaling work loop started event...
[DEBUG] 2022-12-20T08:04:06.0802032-04:00 [] - {BackendCommComponentBase (CentralConfigurationFetcher)} Work loop started signaled
[INFORMATION] 2022-12-20T08:04:07.1741186-04:00 [] - {MetricsCollector} Collecting metrics in 30000 milliseconds interval
[DEBUG] 2022-12-20T08:04:07.2879246-04:00 [] - {HttpDiagnosticListener} Current runtime is not detected as Full Framework - returning implementation for Core. RuntimeInformation.FrameworkDescription: {RuntimeInformation.FrameworkDescription}
[DEBUG] 2022-12-20T08:04:07.3770592-04:00 [] - {DiagnosticInitializer} Subscribed Elastic.Apm.DiagnosticListeners.HttpDiagnosticListenerCoreImpl to `HttpHandlerDiagnosticListener' events source
[DEBUG] 2022-12-20T08:04:07.3895688-04:00 [] - {DiagnosticInitializer} Subscribed Elastic.Apm.SqlClient.SqlClientDiagnosticListener to `SqlClientDiagnosticListener' events source
[DEBUG] 2022-12-20T08:04:07.4643053-04:00 [] - {DiagnosticInitializer} Subscribed Elastic.Apm.SqlClient.SqlClientDiagnosticListener to `SqlClientDiagnosticListener' events source
[FATAL] 2022-12-20T08:04:07.4880027-04:00 [] - Application startup exception
System.MissingFieldException: Field not found: 'Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1.ckingField'.
at Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
at Elasticsearch.Net.Diagnostics.AuditDiagnosticObserver..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
at Elastic.Apm.Elasticsearch.AuditDiagnosticsListener..ctor(IApmAgent agent)
at Elastic.Apm.Elasticsearch.ElasticsearchDiagnosticsSubscriber.Subscribe(IApmAgent agentComponents)
at Elastic.Apm.ApmAgentExtensions.Subscribe(IApmAgent agent, IDiagnosticsSubscriber[] subscribers)
at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, ApmAgent agent, IApmLogger logger, IDiagnosticsSubscriber[] subscribers)
at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, IConfiguration configuration, IDiagnosticsSubscriber[] subscribers)
at Elastic.Apm.NetCoreAll.ApmMiddlewareExtension.UseAllElasticApm(IApplicationBuilder builder, IConfiguration configuration)
at MyProject.Backend.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, ICorsOriginValidator originValidator, ILogger`1 logger) in /src/MyProject.Backend/Startup.cs:line 287
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Unhandled exception. System.MissingFieldException: Field not found: 'Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1.ckingField'.
at Elasticsearch.Net.Diagnostics.TypedDiagnosticObserverBase`1..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
at Elasticsearch.Net.Diagnostics.AuditDiagnosticObserver..ctor(Action`1 onNext, Action`1 onError, Action onCompleted)
at Elastic.Apm.Elasticsearch.AuditDiagnosticsListener..ctor(IApmAgent agent)
at Elastic.Apm.Elasticsearch.ElasticsearchDiagnosticsSubscriber.Subscribe(IApmAgent agentComponents)
at Elastic.Apm.ApmAgentExtensions.Subscribe(IApmAgent agent, IDiagnosticsSubscriber[] subscribers)
at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, ApmAgent agent, IApmLogger logger, IDiagnosticsSubscriber[] subscribers)
at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, IConfiguration configuration, IDiagnosticsSubscriber[] subscribers)
at Elastic.Apm.NetCoreAll.ApmMiddlewareExtension.UseAllElasticApm(IApplicationBuilder builder, IConfiguration configuration)
at MyProject.Backend.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, ICorsOriginValidator originValidator, ILogger`1 logger) in /src/MyProject.Backend/Startup.cs:line 287
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at MyProject.Backend.Program.Main(String[] args) in /src/MyProject.Backend/Program.cs:line 14