Error while restoring

Elastic Stack Elasticsearch
1

Hi Team,

I can see below error while restoring, I took the backup from 2 node cluster and trying to restore in 3 node cluster , i can that restore process hunged and got below errors in logs. Can anyone help me to get rid of this errors.

[ WARN] 2016-02-25 17:51:47,022 org.elasticsearch.indices.cluster - [qa01] [asset_locations][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [asset_locations][3] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [asset_locations][3] restore failed
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:130)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:127)
... 3 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [asset_locations][3] failed to restore snapshot [snapshot-20160225162939]
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:165)
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:124)
... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [asset_locations][3] failed to read shard snapshot file
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:319)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restore(BlobStoreIndexShardRepository.java:709)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:162)
... 5 more
Caused by: java.io.FileNotFoundException: /gdbackup/GDIndexerBackups/director-services/indices/asset_locations/3/snapshot-snapshot-20160225162939 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.openInput(FsBlobContainer.java:79)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:316)
... 7 more
[ WARN] 2016-02-25 17:51:47,027 org.elasticsearch.indices.cluster - [goemtdbpgqa01] [user_profiles][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [user_profiles][3] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [user_profiles][3] restore failed
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:130)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:127)
... 3 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [user_profiles][3] failed to restore snapshot [snapshot-20160225162939]
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:165)
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:124)
... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [user_profiles][3] failed to read shard snapshot file
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:319)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restore(BlobStoreIndexShardRepository.java:709)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:162)
... 5 more
Caused by: java.io.FileNotFoundException: /gdbackup/GDIndexerBackups/director-services/indices/user_profiles/3/snapshot-snapshot-20160225162939 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.openInput(FsBlobContainer.java:79)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.loadSnapshot(BlobStoreIndexShardRepository.java:316)
... 7 more

2

That's why, does that exist on all 3 of the new nodes?

3

The folder /gdbackup/GDIndexerBackups/director-services/indices/asset_locations/3/ is accessible from all the three nodes. But I don’t see the file snapshot-snapshot-20160225162939 in the folder

4

Then that's the issue, where did it go? Is it on the original cluster? What sort of mounts is it, NFS etc?

5

It is a shared NAS storage

6

Well, the data is missing on the new cluster, so you will need to dig into that.

7

Thanks Mark, But the data you are saying is missing in backup data or in the cluster?

8

It's missing from the mount.