Unable to delete snapshot - no space left on device - Solved?

Elastic Stack Elasticsearch
#1

Hi all,

Using 7.10.2 Basic licence

After let my snapshot file system run out of space (100% full), I face the error were no more daily snapshots could run. This file system has nothing but snapshots, so I couldn't delete other files.

Trying to delete some older snapshots also failed with "no space let on device" error.

Adding other disc or somehow increase available space was not an option here, so I pick some
snapshot files to delete. That's what I did:

On configured repository mount point, deleted these two files with system date that match an old snapshot date:

meta-3o1MNGorREq7FuFlDpJ90Q.dat
snap-3o1MNGorREq7FuFlDpJ90Q.dat

and on indices subdirectory, deleted all folders with same system date.

After that, I deleted one more old snapshot, using Kibana's Snapshot and Restore. No errors occurred and both deleted snapshots are no longer listed in Kibana's Snapshot and Restore snapshot list.

Snapshot file system is now 80% used :slight_smile:

Adjust your snapshot Policy (ex. retention days) if needed to avoid future problems.

I'm not sure if this procedure can be replicated or if was just luck to have file system dates that match snapshot dates. Does anyone tried this approach?

Thanks.

#2

It was just luck if this worked: manually deleting files from a snapshot repository can render it completely unreadable. It might bite you in future too, even if it seems to be working now there's no guarantee it'll carry on working in future.

1 Like
#3

I know that's not how this should be done.
Maybe file system space monitoring is the way but I guess Elastic could prevent this to happen by not using all available space on file system, so we could at least start to delete older backups.

Is there any way to know how much space each snapshot takes so we could estimate a new file system size?

Thank you

#4

I'm afraid no such thing exists yet. We have an open issue for implementing functionality for this here:

#5

I opened an issue to suggest that:

1 Like
#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.