Hello,
I upgraded an Elasticsearch cluster from 7.10 to 7.17.9. ES upgrade is fine, with all the nodes up. However, when upgrading Kibana, I got the error when it attempts to re-index.
{"type":"log","@timestamp":"2023-08-08T08:21:18+02:00","tags":["info","savedobjects-service"],"pid":70955,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 120373ms."}
{"type":"log","@timestamp":"2023-08-08T08:23:19+02:00","tags":["error","savedobjects-service"],"pid":70955,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds."}
It goes on in a loop for 20 minutes, then it starts again.
This is another log;
{"type":"log","@timestamp":"2023-08-08T08:16:48+02:00","tags":["info","savedobjects-service"],"pid":69308,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> FATAL. took: 184011ms."}
{"0":"{","1":"\"","2":"s","3":"u","4":"c","5":"c","6":"e","7":"e","8":"d","9":"e","10":"d","11":"\"","12":":","13":"f","14":"a","15":"l","16":"s","17":"e","18":",","19":"\"","20":"n","21":"u","22":"m","23":"_","24":"f","25":"r","26":"e","27":"e","28":"d","29":"\"","30":":","31":"0","32":"}","type":"log","@timestamp":"2023-08-08T08:16:48+02:00","tags":["warning","savedobjects-service"],"pid":69308,"message":"Failed to cleanup after migrations:"}
{"type":"log","@timestamp":"2023-08-08T08:16:48+02:00","tags":["fatal","root"],"pid":69308,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the REINDEX_SOURCE_TO_TEMP_INDEX_BULK step after 15 attempts, terminating.\n at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Promise.all (index 1)\n at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)\n at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)\n at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}
{"type":"log","@timestamp":"2023-08-08T08:16:48+02:00","tags":["info","plugins-system","standard"],"pid":69308,"message":"Stopping all plugins."}