Restore of snapshot from s3 is failing

Elastic Stack Elasticsearch
1

Hi,

I have two elasticsearch clusters (prod and dev). I have both clusters pointing to the same s3 back snapshot repo. The snapshots in prod say they are completing without error. However, when I try to restore from the snapshot to the dev cluster I keep getting the error "no_such_file_exception" "blob object [indices/folder/meta-file.dat]" not found. This is on elasticsearch 8.14. This error occurs no matter which snapshot I select. Has anyone seen this issue before? If the snapshot was created successfully I am not sure why it can't restore. Thanks in advance.
Mark

2

Can you try 8.16.0 instead? This sounds like a bug, but I think 8.16.0 handles it better and should succeed.

3

Hi @DavidTurner ,

Thanks for the quick reply. It turns out I had a OLM rule applied to bucket and I think that deleted some key files. Cleared out that bucket completely, removed the OLM, recreated the repos in elasticsearch and now I can restore. The only issue I am seeing now is sometimes one cluster can see the repo but the other cluster cant. I have to restart all the pods on the cluster that cant access the s3 bucket and then it works. Then the other one cluster loses access. I am trying to use the eks service role of each cluster to access the bucket. That way I don't have to mess with access keys and secrets. Any suggestions on this issue?

Thanks,
Mark

4

Those symptoms suggest that you have not set read_only: true on the cluster that isn't writing to the repository.