Hi all,
We're having fairly frequent issues with our pre-prod environment snapshots failing. We're using Azure as our repo, all nodes can see it and it passes the normal validation steps.
I can only see a couple of lines where the scheduled snapshot ran
[2022-01-31T04:08:04,031][WARN ][o.e.s.SnapshotShardsService] [SERVER-A01] [[wazuh-alerts-4.x-2022.01.26][0]][AzureRepo:testweekly-2022.01.31-h-27i5gar7q_sjaqdft-dw/uNkTzbvySkK2w1Ue620Z8A] failed to snapshot shard
org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
We retried it later today and it completed successfully but there's some warnings, not sure if they have any relevance to the failures?
[2022-01-31T12:51:41,698][WARN ][o.e.r.b.BlobStoreRepository] [SERVER-A01] [AzureRepo] Unknown blob in the repository: index-sXY0MaVZRZm0QfSrG127_w
[2022-01-31T12:51:41,698][WARN ][o.e.r.b.BlobStoreRepository] [SERVER-A01] Could not find a readable index-N file in a non-empty shard snapshot directory [[testsnapshots][indices][MO0xJTo5SSSw6XGfluGBTw][0]]
Here's a longer log file including the traces: [2022-01-31T04:08:04,031][WARN ][o.e.s.SnapshotShardsService] [SERVER-A01] [[waz - Pastebin.com
*Edit to say that it's not always the same shard or indices. Today it's a Wazuh index, but we've see it from WinlogBeats, FileBeat and Metricbeat indices.