# SnapShot Backup and Restore

**URL:** <https://discuss.elastic.co/t/snapshot-backup-and-restore/333943>\
**Category:** Kibana\
**Tags:** elastic-stack-monitoring, elastic-stack-security, elastic-stack-alerting, docker, painless\
**Created:** [May 21, 2023, 10:45am UTC](https://discuss.elastic.co/t/snapshot-backup-and-restore/333943 "2023-05-21T10:45:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manjunath\_VS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manjunath_vs/32/118032_2.png) [@Manjunath\_VS](https://discuss.elastic.co/u/Manjunath_VS)\
**Post date:** [May 21, 2023, 10:45am UTC](https://discuss.elastic.co/t/snapshot-backup-and-restore/333943/1 "2023-05-21T10:45:50Z")

</div>

Hi Team,  
I have deployed Elasticsearch cluster along with Kibana using Bitnami Elasticsearch

> **[charts/bitnami/elasticsearch at main · bitnami/charts](https://github.com/bitnami/charts/tree/main/bitnami/elasticsearch)**
>
> Bitnami Helm Charts. Contribute to bitnami/charts development by creating an account on GitHub.

I am new to Elasticsearch and Kibana  
We wanted to take a backup from a Kubernetes cluster and restore it to another Kubernetes cluster and looking to restore the complete Elasticsearch cluster state along with indices and Kibana Dashboards.

We are using 8.6.2, 8.7.1 and 7.17.9 versions on Kubernetes deployment .

Can you please help us on the right steps and process to follow the backup and restore

I tried copying the entire dir of /bitnami/elasticsearch/data dir from master pod dev cluster to qa cluster master pod on the same dir of qa cluster

Created a Repository on QA Cluster with the same name as created on dev cluster using the same path of the dir  
But I am unable to see the SnapShots or unable to restore the complete cluster state from DEV to QA

# Also on snapshot repository verification status I am seeing these errors

# "suppressed": [{ "type": "repository\_verification\_exception", "reason": "[dca-backup] a file written by master to the store [/bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/dca-backup] cannot be accessed on the node [{dcaw-elastic-master-1}{qCfSq2YnQyKVomObsWciGA}{uX-ca0EXQgGjXwvNGoPGvg}{dcaw-elastic-master-1}{dcaw-elastic-master-1.dcaw-elastic-master-hl.elastic.svc.cluster.local}{172.30.21.252:9300}{m}{xpack.installed=true}]. This might indicate that the store [/bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/dca-backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node"

I tried following few docs from official documents and I am seeing these errors from snapshot and restore options  
2023-05-21T13:26:14,203][WARN][o.e.s.InternalSnapshotsInfoService] [dcaw-elastic-master-0] failed to retrieve shard size for [snapshot=dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ, index=[avg\_by\_dc\_type/cOn-H96iQVGIhKKetNv4wA], shard=[avg\_by\_dc\_type][0]] org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:3223)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:218)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  
at java.base/java.lang.Thread.run(Thread.java:833)  
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/cOn-H96iQVGIhKKetNv4wA/0/snap-wbsV45h\_Rm-8Ts\_ilhazPQ.dat  
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)  
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)  
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)  
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)  
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)  
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)  
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)  
at java.base/java.nio.file.Files.newInputStream(Files.java:160)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)  
... 7 more

[2023-05-21T13:26:14,204][WARN][o.e.s.InternalSnapshotsInfoService] [dcaw-elastic-master-0] failed to retrieve shard size for [snapshot=dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ, index=[chiller\_data/TIAmh96lQ5CMlrO9hLfjtg], shard=[chiller\_data][0]] org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:3223)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:218)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  
at java.base/java.lang.Thread.run(Thread.java:833)  
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/TIAmh96lQ5CMlrO9hLfjtg/0/snap-wbsV45h\_Rm-8Ts\_ilhazPQ.dat  
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)  
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)  
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)  
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)  
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)  
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)  
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)  
at java.base/java.nio.file.Files.newInputStream(Files.java:160)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)  
... 7 more

[2023-05-21T13:26:14,298][WARN][o.e.h.n.Netty4HttpServerTransport] [dcaw-elastic-master-0] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52678}  
[2023-05-21T13:26:14,311][WARN][o.e.h.n.Netty4HttpServerTransport] [dcaw-elastic-master-0] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52682}  
[2023-05-21T13:26:14,389][WARN][o.e.c.r.a.AllocationService] [dcaw-elastic-master-0] failing shard [FailedShard[routingEntry=[chiller\_data][0], node[qlN7dBYxRoyAj-TnYX0o2w], [P], recovery\_source[snapshot recovery [dpav5ZW-SySYl36m23MNTw] from dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ], s[INITIALIZING], a[id=AyOuY-8VQfKu\_ashh-ku4g], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2023-05-21T13:26:14.125Z], failed\_attempts[4], failed\_nodes[[qlN7dBYxRoyAj-TnYX0o2w]], delayed=false, last\_node[qlN7dBYxRoyAj-TnYX0o2w], details[failed shard on node [qlN7dBYxRoyAj-TnYX0o2w]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [chiller\_data][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}  
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)  
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)  
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)  
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)  
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  
at java.lang.Thread.run(Thread.java:833)  
Caused by: [chiller\_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller\_data][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery  
... 14 more  
Caused by: [chiller\_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller\_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed  
... 12 more  
Caused by: [chiller\_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller\_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ]  
... 10 more  
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)  
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
... 3 more  
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/TIAmh96lQ5CMlrO9hLfjtg/0/snap-wbsV45h\_Rm-8Ts\_ilhazPQ.dat  
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)  
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)  
at java.nio.file.Files.newByteChannel(Files.java:380)  
at java.nio.file.Files.newByteChannel(Files.java:432)  
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)  
at java.nio.file.Files.newInputStream(Files.java:160)  
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)  
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)  
... 7 more  
], allocation\_status[fetching\_shard\_data]], failed\_attempts[0], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [chiller\_data][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}, markAsStale=true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [chiller\_data][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}  
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)  
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)  
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)  
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)  
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  
at java.lang.Thread.run(Thread.java:833)  
Caused by: [chiller\_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller\_data][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery  
... 14 more  
Caused by: [chiller\_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller\_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed  
... 12 more  
Caused by: [chiller\_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller\_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ]  
... 10 more  
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)  
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
... 3 more  
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/TIAmh96lQ5CMlrO9hLfjtg/0/snap-wbsV45h\_Rm-8Ts\_ilhazPQ.dat  
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)  
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)  
at java.nio.file.Files.newByteChannel(Files.java:380)  
at java.nio.file.Files.newByteChannel(Files.java:432)  
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)  
at java.nio.file.Files.newInputStream(Files.java:160)  
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)  
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)  
... 7 more

[2023-05-21T13:26:14,389][WARN][o.e.c.r.a.AllocationService] [dcaw-elastic-master-0] failing shard [FailedShard[routingEntry=[avg\_by\_dc\_type][0], node[qlN7dBYxRoyAj-TnYX0o2w], [P], recovery\_source[snapshot recovery [dpav5ZW-SySYl36m23MNTw] from dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ], s[INITIALIZING], a[id=QnNawv4wTUORZXy4uHRjsg], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2023-05-21T13:26:14.124Z], failed\_attempts[4], failed\_nodes[[qlN7dBYxRoyAj-TnYX0o2w]], delayed=false, last\_node[qlN7dBYxRoyAj-TnYX0o2w], details[failed shard on node [qlN7dBYxRoyAj-TnYX0o2w]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [avg\_by\_dc\_type][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}  
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)  
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)  
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)  
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)  
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  
at java.lang.Thread.run(Thread.java:833)  
Caused by: [avg\_by\_dc\_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg\_by\_dc\_type][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery  
... 14 more  
Caused by: [avg\_by\_dc\_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg\_by\_dc\_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed  
... 12 more  
Caused by: [avg\_by\_dc\_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg\_by\_dc\_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ]  
... 10 more  
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)  
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
... 3 more  
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/cOn-H96iQVGIhKKetNv4wA/0/snap-wbsV45h\_Rm-8Ts\_ilhazPQ.dat  
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)  
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)  
at java.nio.file.Files.newByteChannel(Files.java:380)  
at java.nio.file.Files.newByteChannel(Files.java:432)  
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)  
at java.nio.file.Files.newInputStream(Files.java:160)  
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)  
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)  
... 7 more  
], allocation\_status[fetching\_shard\_data]], failed\_attempts[0], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [avg\_by\_dc\_type][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}, markAsStale=true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [avg\_by\_dc\_type][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}  
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)  
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)  
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)  
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)  
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)  
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  
at java.lang.Thread.run(Thread.java:833)  
Caused by: [avg\_by\_dc\_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg\_by\_dc\_type][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery  
... 14 more  
Caused by: [avg\_by\_dc\_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg\_by\_dc\_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed  
... 12 more  
Caused by: [avg\_by\_dc\_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg\_by\_dc\_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ]  
... 10 more  
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)  
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
... 3 more  
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/cOn-H96iQVGIhKKetNv4wA/0/snap-wbsV45h\_Rm-8Ts\_ilhazPQ.dat  
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)  
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)  
at java.nio.file.Files.newByteChannel(Files.java:380)  
at java.nio.file.Files.newByteChannel(Files.java:432)  
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)  
at java.nio.file.Files.newInputStream(Files.java:160)  
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)  
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)  
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)  
... 7 more

[2023-05-21T13:26:14,453][WARN][o.e.s.InternalSnapshotsInfoService] [dcaw-elastic-master-0] failed to retrieve shard size for [snapshot=dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ, index=[avg\_by\_dc\_type/cOn-H96iQVGIhKKetNv4wA], shard=[avg\_by\_dc\_type][0]] org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6\_qvcux4ywca/wbsV45h\_Rm-8Ts\_ilhazPQ] is missing  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:3223)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:218)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)  
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  
at

Regards,

Manjunath VS

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [May 25, 2023, 4:28pm UTC](https://discuss.elastic.co/t/snapshot-backup-and-restore/333943/2 "2023-05-25T16:28:53Z")

</div>

Hi @Manjunath_VS

Snapshots from earlier later versions are not compatible with earlier versions. You don't say specifically which environment is what version of elasticsearch, but you won't be able to use a snapshot from 8.7.1 in 8.6.2 for example.

Please see the notes on snapshots but specifically the part about [restoring data](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html#snapshot-repository-contents).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2023, 4:29pm UTC](https://discuss.elastic.co/t/snapshot-backup-and-restore/333943/3 "2023-06-22T16:29:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.