From last 2 4 weeks, Sanpshots are failing with the following the same error message. Finding a solution for snapshot creation where the same policies were working fine earlier but all of sudden the same policies are working.
Steps followed such as:
Created new policy only data indices are included and run. Still the same issue.
Closed unassigned indices as per the message. Still the same issue.
Deleted unassigned indices as per the message. Still the same issue.
Please find the message that popullated.
{
"type": "snapshot_exception",
"reason": "[fastapibackups:daily-2022-12-09-05-30-15-4vpl2awls9uoeesbmfy1fq/GKK9AppKRD-X1s8cB3vesQ] Indices don't have primary shards [.transform-notifications-000002, .watches, .kibana_task_manager_7.17.0_001]",
"stack_trace": "SnapshotException[[fastapibackups:daily-2022-12-09-05-30-15-4vpl2awls9uoeesbmfy1fq/GKK9AppKRD-X1s8cB3vesQ] Indices don't have primary shards [.transform-notifications-000002, .watches, .kibana_task_manager_7.17.0_001]]\n\tat org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:574)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:487)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:51)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:836)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:403)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)\n\tat org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"
}
Current version ELK 7.17.0
All snapshot drives Free and healthy.
All Nodes are healthy and fine.
And also, even searching for any unassigned index.. using CLI method.. getting the following message every time with changing of different index..
{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index_uuid":"mZvryvwwTdKq66J9ank1Qg","index":".siem-signals-default-000001"}],"type":"index_closed_exception","reason":"closed","index_uuid":"mZvryvwwTdKq66J9ank1Qg","index":".siem-signals-default-000001"},"status":400}