After let my snapshot file system run out of space (100% full), I face the error were no more daily snapshots could run. This file system has nothing but snapshots, so I couldn't delete other files.
Trying to delete some older snapshots also failed with "no space let on device" error.
Adding other disc or somehow increase available space was not an option here, so I pick some
snapshot files to delete. That's what I did:
On configured repository mount point, deleted these two files with system date that match an old snapshot date:
and on indices subdirectory, deleted all folders with same system date.
After that, I deleted one more old snapshot, using Kibana's Snapshot and Restore. No errors occurred and both deleted snapshots are no longer listed in Kibana's Snapshot and Restore snapshot list.
Snapshot file system is now 80% used
Adjust your snapshot Policy (ex. retention days) if needed to avoid future problems.
I'm not sure if this procedure can be replicated or if was just luck to have file system dates that match snapshot dates. Does anyone tried this approach?
It was just luck if this worked: manually deleting files from a snapshot repository can render it completely unreadable. It might bite you in future too, even if it seems to be working now there's no guarantee it'll carry on working in future.
I know that's not how this should be done.
Maybe file system space monitoring is the way but I guess Elastic could prevent this to happen by not using all available space on file system, so we could at least start to delete older backups.
Is there any way to know how much space each snapshot takes so we could estimate a new file system size?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
