Hi,
After updating Elasticsearch and Kibana to 8.9.0, our Alerts stopped working:
I do not understand what is the problem here.
Here is the logs of Kibana:
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-09-29T18:38:47.726+00:00","message":"There has been a catastrophic error trying to install index level resources for the following registration context: observability.logs. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms","error":{"message":"There has been a catastrophic error trying to install index level resources for the following registration context: observability.logs. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms","type":"RuleDataWriterInitializationError","stack_trace":"RuleDataWriterInitializationError: There has been a catastrophic error trying to install index level resources for the following registration context: observability.logs. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms\n at RuleDataClient.initializeWriter (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.js:146:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at RuleDataClient.getWriter (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.js:120:34)\n at Object.executor (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/utils/create_lifecycle_executor.js:37:32)\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:288:26\n at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:50:20)\n at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:266:9)\n at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:594:49)\n at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:274:22)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":669},"trace":{"id":"b7902fb8c8b71a32c298836d6fb035c5"},"transaction":{"id":"933f24d83aa45aa9"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-09-29T18:38:47.727+00:00","message":"The writer for the Rule Data Client for the observability.logs registration context was not initialized properly, bulk() cannot continue.","log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":669},"trace":{"id":"b7902fb8c8b71a32c298836d6fb035c5"},"transaction":{"id":"933f24d83aa45aa9"}}
{"tags":["logs.alert.document.count","a8142c50-b46e-11ed-9b74-ddc1ba307bca","rule-run-failed"],"error":{"stack_trace":"RuleDataWriterInitializationError: There has been a catastrophic error trying to install index level resources for the following registration context: observability.logs. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms\n at RuleDataClient.initializeWriter (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.js:146:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at RuleDataClient.getWriter (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.js:120:34)\n at Object.executor (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/utils/create_lifecycle_executor.js:37:32)\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:288:26\n at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:50:20)\n at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:266:9)\n at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:594:49)\n at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:274:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-09-29T18:38:47.797+00:00","message":"Executing Rule default:logs.alert.document.count:a8142c50-b46e-11ed-9b74-ddc1ba307bca has resulted in Error: There has been a catastrophic error trying to install index level resources for the following registration context: observability.logs. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms - RuleDataWriterInitializationError: There has been a catastrophic error trying to install index level resources for the following registration context: observability.logs. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms\n at RuleDataClient.initializeWriter (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.js:146:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at RuleDataClient.getWriter (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.js:120:34)\n at Object.executor (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/utils/create_lifecycle_executor.js:37:32)\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:288:26\n at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:50:20)\n at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:266:9)\n at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:594:49)\n at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:274:22)","log":{"level":"ERROR","logger":"plugins.alerting.logs.alert.document.count"},"process":{"pid":669},"trace":{"id":"b7902fb8c8b71a32c298836d6fb035c5"},"transaction":{"id":"933f24d83aa45aa9"}}
Please note that observability.uptime seems to be impacted too.
Could you help me find the source of the problem here please?
Thanks in advance.