Curator snapshot list is not accessible after deletion of some files accidentally

[amruta.mohite (Amruta)]

Elasticsearch Snapshot Error

Questions is similar to Error Missing Snapshot Error We investigated into this, looks like upgrade Elasticsearch from 5 to 6 broke it. Is there any way to fix this now? If not able to fix it, we want to use curator to delete some of those old snapshots but unable to do so. Error: Snapshot does not exist. Is there solution to skip it?

Missing Snapshot Error

I'm trying to use curator to delete all snapshots older then a 100 days in my repository. I noticed it was failing with the error message: Failed to complete action: delete_snapshots. <class 'curator.exceptions.FailedExecution'>: Unable to get snapshot information from repository: dcselasticsnapshot. Error: TransportError(500, 'snapshot_exception', '[dcselasticsnapshot:citydirectory_ocrpages/XGftDvzIRPGe_wMfN6FsSw] is missing') Looking into this further, I went and tried to list all snapshots …

I am currently facing same issue in my ELK cluster. The cause which made this happen is I accidentally deleted some files(applied glacier life-cycle policy to delete data after some interval) from S3 bucket and then my curator repository stopped working and hence I am not able to list the snapshots and restore indices as well.
I can't afford to start afresh with new bucket as my application is on production environment and I need data now and then.

Please give me a way so that I could be able to access my repository without losing data.
I tried every possible way to access it like trying to delete that specific snapshot and installing curator cli and adding delete_snapshot in action file but none of this helping.

Please help me as soon as possible.

[theuntergeek (Aaron Mildenstein)]

@amruta.mohite

I accidentally deleted some files(applied glacier life-cycle policy to delete data after some interval) from S3 bucket

I hate to be the bearer of bad news here, but if you accidentally deleted repository metadata, there is no way to get it back of which I am aware.

The metadata that was deleted (not knowing everything missing, but guessing) contained the information concerning which indices contained which segments, and which snapshots contained which indices (and their associated segments). If that data is gone, there is no way to re-create it. Even if the snapshotted data (the index segments themselves) is intact, it is useless without the metadata that allows Elasticsearch to reconstruct them into a viable index.

Now, based on what I'm seeing in this message:

@amruta.mohite

Error: TransportError(500, 'snapshot_exception', '[dcselasticsnapshot:citydirectory_ocrpages/XGftDvzIRPGe_wMfN6FsSw] is missing'

You appear to have deleted some of your segment data. The snapshot metadata may actually be intact, but files it requires to be able to reconstruct certain indices are gone. You can still restore from this repository, but it will be missing some data (how much depends on what, exactly, was deleted). You only need to set partial: true during a restore for this to work. Unfortunately, you will not likely be able to snapshot to this repository in this state. It won't let you because it's not healthy.

Potential solutions include restoring what data you have to a temporary cluster and re-snapshotting it to a re-made repository.

[amruta.mohite (Amruta)]

Thank you for the responding @theuntergeek.
I will create a new repository and the data available will use in next one.

Still one last question, Can we manually delete snapshot metadata of certain age so that it doesn't look for those data anymore?

I have tried with delete snapshot older than specific date but same error I was getting with command, but if any other way manually or through commands, it will be a great help.

[theuntergeek (Aaron Mildenstein)]

@amruta.mohite

Can we manually delete snapshot metadata of certain age so that it doesn't look for those data anymore?

Only by deleting old snapshots will older metadata also be deleted. In your case, because of the missing data, this does not appear to be possible using the approved methods (API calls).