# ElasticSearch fails to backup certain indices
**URL:** https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295
**Category:** Elasticsearch
**Created:** [October 18, 2016, 12:51pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295 "2016-10-18T12:51:06Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@covidium](https://discuss.elastic.co/u/covidium)
#### Post date: [October 18, 2016, 12:51pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/1 "2016-10-18T12:51:06Z")
Hi there,
Before posting this message, I've searched the forum but didn't find anything relevant for the issue I've encountered. But if anyone thinks that the forum has a similar post, please point me to that post, otherwise please continue to read.
Unfortunately I couldn't find any solution to solve this issue, and frankly this is my last option.
This behavior has been noticed on different indices stored in the clusters and have no connection to the date or the size of the indices. I've seen failing indices from 2 days ago with 350KB in size and indices older than 20 days with 14GB in size.
If you need any other details beside those provided bellow, please let me know.
**Issue** : Elasticsearch fails to snapshot certain indices.
**Cause** : Unknown
**Details** :
ES Clusters:
> A) 1 Hot node, 1 Warm node
> B) 3 Hot nodes, 5 Warm nodes
> Both clusters are running 1.6.2
Snapshot storage:
> NFS mount accessible from all the nodes. Path /backups/\*name-of-the-cluster ;
> Available disk space on the backup server 5.2TB
> Available inodes on the backup server 10485688991
Elasticsearch config:
> path:
> data: /usr/share/elasticsearch/data/staging201
> repo: /backups/staging201
Command executed:
> curl -XPUT [http://localhost:9200/\_snapshot/staging201/solrsearch-2016.09.26?wait\_for\_completion=true](http://localhost:9200/_snapshot/staging201/solrsearch-2016.09.26?wait_for_completion=true) -d '{"indices":"solrsearch-2016.09.26", "ignore\_unavailable": "true", "include\_global\_state": false}'
Output on the console:
> {"snapshot":{"snapshot":"solrsearch-2016.09.26","indices":["solrsearch-2016.09.26"],"state":"PARTIAL","start\_time":"2016-10-18T11:03:50.867Z","start\_time\_in\_millis":1476788630867,"end\_time":"2016-10-18T11:04:00.477Z","end\_time\_in\_millis":1476788640477,"duration\_in\_millis":9610,"failures":[{"node\_id":"ytzwGkYJQqWqvpJ\_fkv3kQ","index":"solrsearch-2016.09.26","reason":"IndexShardSnapshotFailedException[[solrsearch-2016.09.26][2] failed to list blobs]; nested: NoSuchFileException[/backups/staging201/indices/solrsearch-2016.09.26/2]; ","shard\_id":2,"status":"INTERNAL\_SERVER\_ERROR"},{"node\_id":"ytzwGkYJQqWqvpJ\_fkv3kQ","index":"solrsearch-2016.09.26","reason":"IndexShardSnapshotFailedException[[solrsearch-2016.09.26][1] failed to list blobs]; nested: NoSuchFileException[/backups/staging201/indices/solrsearch-2016.09.26/1]; ","shard\_id":1,"status":"INTERNAL\_SERVER\_ERROR"},{"node\_id":"ytzwGkYJQqWqvpJ\_fkv3kQ","index":"solrsearch-2016.09.26","reason":"IndexShardSnapshotFailedException[[solrsearch-2016.09.26][3] failed to list blobs]; nested: NoSuchFileException[/backups/staging201/indices/solrsearch-2016.09.26/3]; ","shard\_id":3,"status":"INTERNAL\_SERVER\_ERROR"}],"shards":{"total":5,"failed":3,"successful":2}}}
\_
Messages in log file:
> SEE NEXT MESSAGE
---
### Author:  [@covidium](https://discuss.elastic.co/u/covidium)
#### Post date: [October 18, 2016, 12:52pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/2 "2016-10-18T12:52:45Z")
Messages in log file:
> [2016-10-18 13:03:50,925][WARN][snapshots] [cr1-elk-211-staging201] [[solrsearch-2016.09.26][2]] [staging201:solrsearch-2016.09.26] failed to create snapshot
> org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [solrsearch-2016.09.26][2] failed to list blobs
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:442)
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:140)
> at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:85)
> at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:871)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.file.NoSuchFileException: /backups/staging201/indices/solrsearch-2016.09.26/2
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
> at java.nio.file.Files.newDirectoryStream(Files.java:457)
> at java.nio.file.Files.newDirectoryStream(Files.java:514)
> at org.elasticsearch.common.blobstore.fs.FsBlobContainer.listBlobsByPrefix(FsBlobContainer.java:65)
> at org.elasticsearch.common.blobstore.fs.FsBlobContainer.listBlobs(FsBlobContainer.java:56)
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:440)
> ... 6 more
> [2016-10-18 13:03:50,956][WARN][snapshots] [cr1-elk-211-staging201] [[solrsearch-2016.09.26][1]] [staging201:solrsearch-2016.09.26] failed to create snapshot
> org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [solrsearch-2016.09.26][1] failed to list blobs
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:442)
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:140)
> at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:85)
> at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:871)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.file.NoSuchFileException: /backups/staging201/indices/solrsearch-2016.09.26/1
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
> at java.nio.file.Files.newDirectoryStream(Files.java:457)
> at java.nio.file.Files.newDirectoryStream(Files.java:514)
> at org.elasticsearch.common.blobstore.fs.FsBlobContainer.listBlobsByPrefix(FsBlobContainer.java:65)
> at org.elasticsearch.common.blobstore.fs.FsBlobContainer.listBlobs(FsBlobContainer.java:56)
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:440)
> ... 6 more
> [2016-10-18 13:03:50,969][WARN][snapshots] [cr1-elk-211-staging201] [[solrsearch-2016.09.26][3]] [staging201:solrsearch-2016.09.26] failed to create snapshot
> org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [solrsearch-2016.09.26][3] failed to list blobs
> [ at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:442)
> at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:140)
> at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:85)
> at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:871)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.file.NoSuchFileException: /backups/staging201/indices/solrsearch-2016.09.26/3
> ...truncated...
---
### Author:  [@covidium](https://discuss.elastic.co/u/covidium)
#### Post date: [October 25, 2016, 7:34am UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/3 "2016-10-25T07:34:15Z")
Anyone?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 25, 2016, 8:34am UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/4 "2016-10-25T08:34:10Z")
> [@covidium](#):
>
> NoSuchFileException: /backups/staging201/indices/solrsearch-2016.09.26/2
What are the permissions on the mount?
---
### Author:  [@covidium](https://discuss.elastic.co/u/covidium)
#### Post date: [October 25, 2016, 9:08am UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/5 "2016-10-25T09:08:36Z")
The solrsearch-2016.09.26 indice has 755 root:root
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 25, 2016, 9:10am UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/6 "2016-10-25T09:10:02Z")
What about the rest of it though?
---
### Author:  [@covidium](https://discuss.elastic.co/u/covidium)
#### Post date: [October 25, 2016, 9:12am UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/7 "2016-10-25T09:12:00Z")
They all have the same permissions and ownership.
What I found strange is that only for certain indices the snapshot fails with this INTERNAL\_SERVER\_ERROR message.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:10pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-backup-certain-indices/63295/8 "2017-07-05T22:10:05Z")