Delete snapshot fails due to no space on disk

Elastic Stack Elasticsearch
1

We run ES 5.2.2 on a 8 node CentOS cluster. We are using filesystem repository to store snapshots. The disk volume storing the snapshot repository ran out of space and is now at 100% used.

We tried running the delete snapshot API to remove older snapshots, but that too fails with the error:

DELETE /_snapshot/repo_01/old_snapshot_01

{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[repo_01] failed to update snapshot in repository"
      }
    ],
    "type": "repository_exception",
    "reason": "[repo_01] failed to update snapshot in repository",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "No space left on device"
    }
  },
  "status": 500
}

I saw this open issue and it appears that ES tries to create file(s) to initiate the delete, and that operation fails as there is no more space available.

Is there any way to delete older snapshots through the API in such a scenario? If not through the API, is there a safe way to figure out which files from older snapshots are no longer needed and delete a few of them them manually from disk, just enough to make space for the delete API to work?

2

It appears there is no way to run the delete API except to somehow clear just enough space on the disk. I managed to figure out some unused files and was then able to run the delete API .

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.