# Unable to delete snapshot - no space left on device - Solved? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 20, 2021, 3:57pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675 "2021-01-20T15:57:19Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.elastic.co/u/mcosta)\ **Post date:** [January 20, 2021, 3:57pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675/1 "2021-01-20T15:57:20Z")
Hi all, Using 7.10.2 Basic licence After let my snapshot file system run out of space (100% full), I face the error were no more daily snapshots could run. This file system has nothing but snapshots, so I couldn't delete other files. Trying to delete some older snapshots also failed with "no space let on device" error. Adding other disc or somehow increase available space was not an option here, so I pick some snapshot files to delete. That's what I did: On configured repository mount point, deleted these two files with system date that match an old snapshot date: meta-3o1MNGorREq7FuFlDpJ90Q.dat snap-3o1MNGorREq7FuFlDpJ90Q.dat and on indices subdirectory, deleted all folders with same system date. After that, I deleted one more old snapshot, using Kibana's Snapshot and Restore. No errors occurred and both deleted snapshots are no longer listed in Kibana's Snapshot and Restore snapshot list. Snapshot file system is now 80% used 🙂 Adjust your snapshot Policy (ex. retention days) if needed to avoid future problems. I'm not sure if this procedure can be replicated or if was just luck to have file system dates that match snapshot dates. Does anyone tried this approach? Thanks. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 20, 2021, 4:56pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675/2 "2021-01-20T16:56:35Z")
> [@mcosta](#): > > I'm not sure if this procedure can be replicated or if was just luck to have file system dates that match snapshot dates. Does anyone tried this approach? It was just luck if this worked: manually deleting files from a snapshot repository can render it completely unreadable. It might bite you in future too, even if it seems to be working now there's no guarantee it'll carry on working in future. ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.elastic.co/u/mcosta)\ **Post date:** [January 20, 2021, 7:29pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675/3 "2021-01-20T19:29:23Z")
I know that's not how this should be done. Maybe file system space monitoring is the way but I guess Elastic could prevent this to happen by not using all available space on file system, so we could at least start to delete older backups. Is there any way to know how much space each snapshot takes so we could estimate a new file system size? Thank you ---
**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\ **Post date:** [January 20, 2021, 7:46pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675/4 "2021-01-20T19:46:29Z")
> [@mcosta](#): > > Is there any way to know how much space each snapshot takes so we could estimate a new file system size? I'm afraid no such thing exists yet. We have an open issue for implementing functionality for this here: > > > We've had various requests for a way of determining the storage savings that cou…ld be had from deleting a snapshot before deleting it. Also, we need a way to determine the size of a snapshot repository via an API call. > > After discussing this during snapshot resiliency sync we decided that both things could be had by adding a dry-run API for snapshot deletes that returns the bytes that could be saved for a given delete request. This would allow getting the incremental size of a snapshot when dry-running single snapshot deletes as well as getting the size of the full repository by dry-running a delete for all snapshots via snapshot name \`\*\`. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 20, 2021, 7:50pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675/5 "2021-01-20T19:50:14Z")
> [@mcosta](#): > > Maybe file system space monitoring is the way but I guess Elastic could prevent this to happen by not using all available space on file system, so we could at least start to delete older backups. I opened an issue to suggest that: > > > A user \[reported running out of disk space\](https://discuss.elastic.co/t/unable-…to-delete-snapshot-no-space-left-on-device-solved/261675) in their shared filesystem repository which left it completely stuck, unable to take any further actions since everything that might delete any existing data (even repository cleanup AFAICT) starts by writing another metadata file to the repository before proceeding and there wasn't even the space to do that. > > Perhaps we should refuse to write data blobs (but not metadata blobs) to a shared filesystem repository when it is nearly full, leaving at least a few MB of wiggle room for cleanup and recovery from filling up the disk. > > \---- > > \### Workaround > > 1. When space runs out: > a. disable SLM > b. ensure there are no ongoing snapshots > c. extend the filesystem that contains the repo by 100MiB or so > d. delete some snapshots to free up space > e. shrink the filesystem to its original size. > > \#### Alternative workaround > > 1. Ahead of time, create a ~100MiB file in the same filesystem as the repo to reserve some space. > 2. When space runs out: > a. disable SLM > b. ensure there are no ongoing snapshots > c. delete the reserved-space file created in step 1 > d. delete some snapshots to free up space > e. create another ~100MiB reserved-space file ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 17, 2021, 7:50pm UTC](https://discuss.elastic.co/t/unable-to-delete-snapshot-no-space-left-on-device-solved/261675/6 "2021-02-17T19:50:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.