Hi,
Trying to restore the snapshot of an index stored in S3. When I check the status of the snapshot I see this:
curl -X GET "localhost:9200/_snapshot/website/2020-2-24_0-0-2/_status?pretty"
{
"snapshots" : [
{
"snapshot" : "2020-2-24_0-0-2",
"repository" : "website",
"uuid" : "1vvYHnb8QompMOiUsfxH2A",
"state" : "SUCCESS",
"shards_stats" : {
"initializing" : 0,
"started" : 0,
"finalizing" : 0,
"done" : 0,
"failed" : 1,
"total" : 1
},
"stats" : {
"number_of_files" : 0,
"processed_files" : 0,
"total_size_in_bytes" : 0,
"processed_size_in_bytes" : 0,
"start_time_in_millis" : 0,
"time_in_millis" : 0
},
"indices" : {
"honeypot" : {
"shards_stats" : {
"initializing" : 0,
"started" : 0,
"finalizing" : 0,
"done" : 0,
"failed" : 1,
"total" : 1
},
"stats" : {
"number_of_files" : 0,
"processed_files" : 0,
"total_size_in_bytes" : 0,
"processed_size_in_bytes" : 0,
"start_time_in_millis" : 0,
"time_in_millis" : 0
},
"shards" : {
"0" : {
"stage" : "FAILURE",
"stats" : {
"number_of_files" : 0,
"processed_files" : 0,
"total_size_in_bytes" : 0,
"processed_size_in_bytes" : 0,
"start_time_in_millis" : 0,
"time_in_millis" : 0
},
"reason" : "IndexShardSnapshotFailedException[error deleting index file [index-5] during cleanup]; nested: IOException[Exception when deleting blob [index-5]]; nested: AmazonS3Exception[Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 01FA4C0C880EA276)]; "
}
}
}
}
}
]
}
When I check the S3 bucket where the snapshots are stored, I see that the files are there, the filesize of some of them are big, gives me the impression that snapshot of the index was done, but the cleanup process failed. I suspect this because the AWS key that we were using for the S3 plugin was able to add and read files, but not to delete. So during the snapshot process, if ES was trying to delete some files for cleaning up, it would fail.
When I try to restore the snapshot, it gives an error saying snapshot is failed, and it cannot restore a partial snapshot. When I try partial=true, it also doesn't restore anything.
So the question is, is there any chance to recover the data in those snapshot files manually maybe, or using some command-line tool? I see that the snapshots somehow organized, there is indices folder, under there is an ID the same way ES gives id to indices (I assume it is the id of the index that I try to restore, I don't know the id of my index as I lost the cluster, unfortunately), then there are lots of metafiles (binary), one for each snapshot it looks, then node 0 (I was using one node only), and finally a lot of files starts with a double underscore (__0, __1, etc). Some of those files are 1gb for example.
On my cluster, I lost the index I'm trying to restore, but I can see the snapshot status still, as above.
Any idea, any help?
Thx in advance!
Ahmet