Curator snapshots error: Failed to verify all nodes have repository access

Elastic Stack Elasticsearch
1

Hey guys,

I'm trying to snapshot my ES indexes using curator and I'm getting the following error:

/usr/bin/curator snapshot --repository jf_backup --prefix jokefire-  --ignore_unavailable --partial indices --all-indices
2015-10-08 20:42:51,536 INFO      Job starting: snapshot indices
2015-10-08 20:42:51,536 WARNING   Overriding default connection timeout.  New timeout: 21600
2015-10-08 20:42:51,574 INFO      Matching all indices. Ignoring flags other than --exclude.
**2015-10-08 20:42:52,320 ERROR     Failed to verify all nodes have repository access.**
2015-10-08 20:42:52,320 WARNING   Job did not complete successfully.

So then I deleted the contents of the backup repository, then I tried again:

#/usr/bin/curator snapshot --repository jf_backup --prefix jokefire-  --ignore_unavailable --partial indices --all-indices
2015-10-08 20:46:40,527 INFO      Job starting: snapshot indices
2015-10-08 20:46:40,527 WARNING   Overriding default connection timeout.  New timeout: 21600
2015-10-08 20:46:40,554 INFO      Matching all indices. Ignoring flags other than --exclude.
2015-10-08 20:46:41,453 INFO      Snapshot name: jokefire-20151009004641
2015-10-08 20:49:53,275 ERROR     Client raised a TransportError.
2015-10-08 20:49:53,275 WARNING   Job did not complete successfully.

And I'm seeing errors like this in the elasticsearch logs:

[2015-10-08 16:35:39,496][INFO ][snapshots                ] [JF_ES1] snapshot [jf_backup:curator-20151008203219] is done
[2015-10-08 16:35:39,649][WARN ][snapshots                ] [JF_ES1] [jf_backup:curator-20151008203219] failed to finalize snapshot
org.elasticsearch.repositories.RepositoryException: [jf_backup] failed to update snapshot in repository
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:365)
        at org.elasticsearch.snapshots.SnapshotsService$7.run(SnapshotsService.java:1104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
        at java.io.FileOutputStream.close0(Native Method)
        at java.io.FileOutputStream.access$000(FileOutputStream.java:53)
        at java.io.FileOutputStream$1.close(FileOutputStream.java:356)
        at java.io.FileDescriptor.closeAll(FileDescriptor.java:212)
        at java.io.FileOutputStream.close(FileOutputStream.java:354)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$1.close(FsBlobContainer.java:100)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeSnapshotList(BlobStoreRepository.java:631)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:362)
        ... 4 more

Does anyone know what these errors means and how to fix it? I can't post any more logs because of the character limit on the post.

Thanks

2

Is why

3

Ah! Ok. Got it. Thanks!!!