Curator failed to delete snapshot: No space left on device

Elastic Stack Elasticsearch
1

Hi,
I am facing below issue in Curator snapshot deletion. Curator version- 5.8.1, ES- 7.10.2

2021-09-19 02:01:01,159 ERROR     Failed to complete action: delete_snapshots.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'file_system_exception', '/elasticsearch-backup/backup/pending-index-26-oa9UAjzQR5KCaY7VNe8JuQ: No space left on device')

I saw some discussions on this issue https://github.com/elastic/elasticsearch/issues/67790

I am stuck here. Any suggestions would be helpful.

2

So, Curator is essentially forwarding the 500 error from Elasticsearch here. If you had made the REST API call to delete the snapshot directly, you would encounter the same error.

That said, I'm not sure what to do here beside manually deleting files (which is not great, as it means the metadata would wind up corrupted). As the mount point is named elasticsearch-backup, is this an NFS share? Is there any way to grow that filesystem?

3

Hi,
Thank you for the response. Yes, currently I have increased the size of Elasticsearch-backup storage.