Hi Team,
I have deployed Elasticsearch cluster along with Kibana using Bitnami Elasticsearch
I am new to Elasticsearch and Kibana
We wanted to take a backup from a Kubernetes cluster and restore it to another Kubernetes cluster and looking to restore the complete Elasticsearch cluster state along with indices and Kibana Dashboards.
We are using 8.6.2, 8.7.1 and 7.17.9 versions on Kubernetes deployment .
Can you please help us on the right steps and process to follow the backup and restore
I tried copying the entire dir of /bitnami/elasticsearch/data dir from master pod dev cluster to qa cluster master pod on the same dir of qa cluster
Created a Repository on QA Cluster with the same name as created on dev cluster using the same path of the dir
But I am unable to see the SnapShots or unable to restore the complete cluster state from DEV to QA
Also on snapshot repository verification status I am seeing these errors
"suppressed": [
{
"type": "repository_verification_exception",
"reason": "[dca-backup] a file written by master to the store [/bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/dca-backup] cannot be accessed on the node [{dcaw-elastic-master-1}{qCfSq2YnQyKVomObsWciGA}{uX-ca0EXQgGjXwvNGoPGvg}{dcaw-elastic-master-1}{dcaw-elastic-master-1.dcaw-elastic-master-hl.elastic.svc.cluster.local}{172.30.21.252:9300}{m}{xpack.installed=true}]. This might indicate that the store [/bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/dca-backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node"
I tried following few docs from official documents and I am seeing these errors from snapshot and restore options
2023-05-21T13:26:14,203][WARN ][o.e.s.InternalSnapshotsInfoService] [dcaw-elastic-master-0] failed to retrieve shard size for [snapshot=dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ, index=[avg_by_dc_type/cOn-H96iQVGIhKKetNv4wA], shard=[avg_by_dc_type][0]] org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:3223)
at org.elasticsearch.server@8.7.1/org.elasticsearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:218)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/cOn-H96iQVGIhKKetNv4wA/0/snap-wbsV45h_Rm-8Ts_ilhazPQ.dat
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.base/java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)
... 7 more
[2023-05-21T13:26:14,204][WARN ][o.e.s.InternalSnapshotsInfoService] [dcaw-elastic-master-0] failed to retrieve shard size for [snapshot=dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ, index=[chiller_data/TIAmh96lQ5CMlrO9hLfjtg], shard=[chiller_data][0]] org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:3223)
at org.elasticsearch.server@8.7.1/org.elasticsearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:218)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/TIAmh96lQ5CMlrO9hLfjtg/0/snap-wbsV45h_Rm-8Ts_ilhazPQ.dat
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.base/java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)
... 7 more
[2023-05-21T13:26:14,298][WARN ][o.e.h.n.Netty4HttpServerTransport] [dcaw-elastic-master-0] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52678}
[2023-05-21T13:26:14,311][WARN ][o.e.h.n.Netty4HttpServerTransport] [dcaw-elastic-master-0] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52682}
[2023-05-21T13:26:14,389][WARN ][o.e.c.r.a.AllocationService] [dcaw-elastic-master-0] failing shard [FailedShard[routingEntry=[chiller_data][0], node[qlN7dBYxRoyAj-TnYX0o2w], [P], recovery_source[snapshot recovery [dpav5ZW-SySYl36m23MNTw] from dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ], s[INITIALIZING], a[id=AyOuY-8VQfKu_ashh-ku4g], unassigned_info[[reason=ALLOCATION_FAILED], at[2023-05-21T13:26:14.125Z], failed_attempts[4], failed_nodes[[qlN7dBYxRoyAj-TnYX0o2w]], delayed=false, last_node[qlN7dBYxRoyAj-TnYX0o2w], details[failed shard on node [qlN7dBYxRoyAj-TnYX0o2w]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [chiller_data][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
Caused by: [chiller_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller_data][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
... 14 more
Caused by: [chiller_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
... 12 more
Caused by: [chiller_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ]
... 10 more
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
... 3 more
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/TIAmh96lQ5CMlrO9hLfjtg/0/snap-wbsV45h_Rm-8Ts_ilhazPQ.dat
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.nio.file.Files.newByteChannel(Files.java:380)
at java.nio.file.Files.newByteChannel(Files.java:432)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)
... 7 more
], allocation_status[fetching_shard_data]], failed_attempts[0], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [chiller_data][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}, markAsStale=true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [chiller_data][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
Caused by: [chiller_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller_data][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
... 14 more
Caused by: [chiller_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
... 12 more
Caused by: [chiller_data/iu7ibTsASrm1VcnEoG2PCQ][[chiller_data][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ]
... 10 more
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
... 3 more
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/TIAmh96lQ5CMlrO9hLfjtg/0/snap-wbsV45h_Rm-8Ts_ilhazPQ.dat
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.nio.file.Files.newByteChannel(Files.java:380)
at java.nio.file.Files.newByteChannel(Files.java:432)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)
... 7 more
[2023-05-21T13:26:14,389][WARN ][o.e.c.r.a.AllocationService] [dcaw-elastic-master-0] failing shard [FailedShard[routingEntry=[avg_by_dc_type][0], node[qlN7dBYxRoyAj-TnYX0o2w], [P], recovery_source[snapshot recovery [dpav5ZW-SySYl36m23MNTw] from dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ], s[INITIALIZING], a[id=QnNawv4wTUORZXy4uHRjsg], unassigned_info[[reason=ALLOCATION_FAILED], at[2023-05-21T13:26:14.124Z], failed_attempts[4], failed_nodes[[qlN7dBYxRoyAj-TnYX0o2w]], delayed=false, last_node[qlN7dBYxRoyAj-TnYX0o2w], details[failed shard on node [qlN7dBYxRoyAj-TnYX0o2w]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [avg_by_dc_type][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
Caused by: [avg_by_dc_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg_by_dc_type][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
... 14 more
Caused by: [avg_by_dc_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg_by_dc_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
... 12 more
Caused by: [avg_by_dc_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg_by_dc_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ]
... 10 more
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
... 3 more
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/cOn-H96iQVGIhKKetNv4wA/0/snap-wbsV45h_Rm-8Ts_ilhazPQ.dat
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.nio.file.Files.newByteChannel(Files.java:380)
at java.nio.file.Files.newByteChannel(Files.java:432)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)
... 7 more
], allocation_status[fetching_shard_data]], failed_attempts[0], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [avg_by_dc_type][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}, markAsStale=true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [avg_by_dc_type][0]: Recovery failed on {dcaw-elastic-data-0}{qlN7dBYxRoyAj-TnYX0o2w}{0U6cqWFBRIavZVwrG0X5xg}{dcaw-elastic-data-0}{dcaw-elastic-data-0.dcaw-elastic-data-hl.elastic.svc.cluster.local}{172.30.240.228:9300}{d}{8.7.1}{xpack.installed=true}
at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$23(IndexShard.java:3147)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$7(StoreRecovery.java:393)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$11(StoreRecovery.java:528)
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:175)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:3014)
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:450)
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:941)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
Caused by: [avg_by_dc_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg_by_dc_type][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
... 14 more
Caused by: [avg_by_dc_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg_by_dc_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
... 12 more
Caused by: [avg_by_dc_type/QZ5Q1vqtT2q2Ru5-vUXV8A][[avg_by_dc_type][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ]
... 10 more
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$80(BlobStoreRepository.java:3042)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
... 3 more
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/dcaw-backup/dca-so-backup/indices/cOn-H96iQVGIhKKetNv4wA/0/snap-wbsV45h_Rm-8Ts_ilhazPQ.dat
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.nio.file.Files.newByteChannel(Files.java:380)
at java.nio.file.Files.newByteChannel(Files.java:432)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:205)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3390)
... 7 more
[2023-05-21T13:26:14,453][WARN ][o.e.s.InternalSnapshotsInfoService] [dcaw-elastic-master-0] failed to retrieve shard size for [snapshot=dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ, index=[avg_by_dc_type/cOn-H96iQVGIhKKetNv4wA], shard=[avg_by_dc_type][0]] org.elasticsearch.snapshots.SnapshotMissingException: [dca-so-backup:weeksnapshot-tbrqktkhrw6_qvcux4ywca/wbsV45h_Rm-8Ts_ilhazPQ] is missing
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3392)
at org.elasticsearch.server@8.7.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:3223)
at org.elasticsearch.server@8.7.1/org.elasticsearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:218)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
at org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
Regards,
Manjunath VS