# Error while restoring

**URL:** https://discuss.elastic.co/t/error-while-restoring/43020
**Category:** Elasticsearch
**Created:** [February 29, 2016, 4:42pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020 "2016-02-29T16:42:52Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![vinod\_rao](https://avatars.discourse-cdn.com/v4/letter/v/3e96dc/32.png) [@vinod\_rao](https://discuss.elastic.co/u/vinod_rao)
#### Post date: [February 29, 2016, 4:42pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/1 "2016-02-29T16:42:52Z")

</div>

Hi Team,

I can see below error while restoring, I took the backup from 2 node cluster and trying to restore in 3 node cluster , i can that restore process hunged and got below errors in logs. Can anyone help me to get rid of this errors.

[WARN] 2016-02-25 17:51:47,022 org.elasticsearch.indices.cluster - [qa01] [asset\_locations][3] failed to start shard  
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [asset\_locations][3] failed recovery  
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)  
at java.lang.Thread.run(Unknown Source)  
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [asset\_locations][3] restore failed  
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:130)  
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:127)  
... 3 more  
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [asset\_locations][3] failed to restore snapshot [snapshot-20160225162939]  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:165)  
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:124)  
... 4 more  
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [asset\_locations][3] failed to read shard snapshot file  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:319)  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restore(BlobStoreIndexShardRepository.java:709)  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:162)  
... 5 more  
Caused by: java.io.FileNotFoundException: /gdbackup/GDIndexerBackups/director-services/indices/asset\_locations/3/snapshot-snapshot-20160225162939 (No such file or directory)  
at java.io.FileInputStream.open(Native Method)  
at java.io.FileInputStream.(Unknown Source)  
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.openInput(FsBlobContainer.java:79)  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:316)  
... 7 more  
[WARN] 2016-02-25 17:51:47,027 org.elasticsearch.indices.cluster - [goemtdbpgqa01] [user\_profiles][3] failed to start shard  
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [user\_profiles][3] failed recovery  
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)  
at java.lang.Thread.run(Unknown Source)  
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [user\_profiles][3] restore failed  
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:130)  
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:127)  
... 3 more  
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [user\_profiles][3] failed to restore snapshot [snapshot-20160225162939]  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:165)  
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:124)  
... 4 more  
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [user\_profiles][3] failed to read shard snapshot file  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:319)  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restore(BlobStoreIndexShardRepository.java:709)  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:162)  
... 5 more  
Caused by: java.io.FileNotFoundException: /gdbackup/GDIndexerBackups/director-services/indices/user\_profiles/3/snapshot-snapshot-20160225162939 (No such file or directory)  
at java.io.FileInputStream.open(Native Method)  
at java.io.FileInputStream.(Unknown Source)  
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.openInput(FsBlobContainer.java:79)  
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:316)  
... 7 more

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 29, 2016, 10:26pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/2 "2016-02-29T22:26:29Z")

</div>

> [@vinod\_rao](#):
>
> Caused by: java.io.FileNotFoundException: /gdbackup/GDIndexerBackups/director-services/indices/asset\_locations/3/snapshot-snapshot-20160225162939 (No such file or directory)

That's why, does that exist on all 3 of the new nodes?

---

<div class="post-metadata">

### Author: ![vinod\_rao](https://avatars.discourse-cdn.com/v4/letter/v/3e96dc/32.png) [@vinod\_rao](https://discuss.elastic.co/u/vinod_rao)
#### Post date: [February 29, 2016, 10:44pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/3 "2016-02-29T22:44:17Z")

</div>

> [@warkolm](#):
>
> Caused by: java.io.FileNotFoundException: /gdbackup/GDIndexerBackups/director-services/indices/asset\_locations/3/snapshot-snapshot-20160225162939 (No such file or directory)

The folder /gdbackup/GDIndexerBackups/director-services/indices/asset\_locations/3/ is accessible from all the three nodes. But I don’t see the file snapshot-snapshot-20160225162939 in the folder

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 29, 2016, 10:52pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/4 "2016-02-29T22:52:05Z")

</div>

Then that's the issue, where did it go? Is it on the original cluster? What sort of mounts is it, NFS etc?

---

<div class="post-metadata">

### Author: ![vinod\_rao](https://avatars.discourse-cdn.com/v4/letter/v/3e96dc/32.png) [@vinod\_rao](https://discuss.elastic.co/u/vinod_rao)
#### Post date: [March 1, 2016, 2:51am UTC](https://discuss.elastic.co/t/error-while-restoring/43020/5 "2016-03-01T02:51:47Z")

</div>

It is a shared NAS storage

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 1, 2016, 2:59am UTC](https://discuss.elastic.co/t/error-while-restoring/43020/6 "2016-03-01T02:59:41Z")

</div>

Well, the data is missing on the new cluster, so you will need to dig into that.

---

<div class="post-metadata">

### Author: ![vinod\_rao](https://avatars.discourse-cdn.com/v4/letter/v/3e96dc/32.png) [@vinod\_rao](https://discuss.elastic.co/u/vinod_rao)
#### Post date: [March 1, 2016, 3:16pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/7 "2016-03-01T15:16:20Z")

</div>

Thanks Mark, But the data you are saying is missing in backup data or in the cluster?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 1, 2016, 8:41pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/8 "2016-03-01T20:41:44Z")

</div>

It's missing from the mount.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:12pm UTC](https://discuss.elastic.co/t/error-while-restoring/43020/9 "2017-07-05T23:12:19Z")

</div>


