Hey guys,
I'm trying to snapshot my ES indexes using curator and I'm getting the following error:
/usr/bin/curator snapshot --repository jf_backup --prefix jokefire- --ignore_unavailable --partial indices --all-indices
2015-10-08 20:42:51,536 INFO Job starting: snapshot indices
2015-10-08 20:42:51,536 WARNING Overriding default connection timeout. New timeout: 21600
2015-10-08 20:42:51,574 INFO Matching all indices. Ignoring flags other than --exclude.
**2015-10-08 20:42:52,320 ERROR Failed to verify all nodes have repository access.**
2015-10-08 20:42:52,320 WARNING Job did not complete successfully.
So then I deleted the contents of the backup repository, then I tried again:
#/usr/bin/curator snapshot --repository jf_backup --prefix jokefire- --ignore_unavailable --partial indices --all-indices
2015-10-08 20:46:40,527 INFO Job starting: snapshot indices
2015-10-08 20:46:40,527 WARNING Overriding default connection timeout. New timeout: 21600
2015-10-08 20:46:40,554 INFO Matching all indices. Ignoring flags other than --exclude.
2015-10-08 20:46:41,453 INFO Snapshot name: jokefire-20151009004641
2015-10-08 20:49:53,275 ERROR Client raised a TransportError.
2015-10-08 20:49:53,275 WARNING Job did not complete successfully.
And I'm seeing errors like this in the elasticsearch logs:
[2015-10-08 16:35:39,496][INFO ][snapshots ] [JF_ES1] snapshot [jf_backup:curator-20151008203219] is done
[2015-10-08 16:35:39,649][WARN ][snapshots ] [JF_ES1] [jf_backup:curator-20151008203219] failed to finalize snapshot
org.elasticsearch.repositories.RepositoryException: [jf_backup] failed to update snapshot in repository
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:365)
at org.elasticsearch.snapshots.SnapshotsService$7.run(SnapshotsService.java:1104)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.close0(Native Method)
at java.io.FileOutputStream.access$000(FileOutputStream.java:53)
at java.io.FileOutputStream$1.close(FileOutputStream.java:356)
at java.io.FileDescriptor.closeAll(FileDescriptor.java:212)
at java.io.FileOutputStream.close(FileOutputStream.java:354)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer$1.close(FsBlobContainer.java:100)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeSnapshotList(BlobStoreRepository.java:631)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:362)
... 4 more
Does anyone know what these errors means and how to fix it? I can't post any more logs because of the character limit on the post.
Thanks