Hello,
I currently have 1 master node and 1 data node running in the same datacenter communicating over internal networking. When I try to create a snapshot from the master node, it gives me a internal server error with the reason node shutdown on the index I'm trying to snapshot. I looked into it and the data node's elasticsearch crashed.
This is the log from the Data node
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-07-10T05:39:03,321][WARN ][o.e.s.SnapshotShardsService] [data-1] [[main][3]] [ultimate:athena/zADfqeXBSY6YnEWJu0FF4g] failed to create snapshot
org.elasticsearch.repositories.RepositoryMissingException: [ultimate] missing
at org.elasticsearch.repositories.RepositoriesService.repository(RepositoriesService.java:334) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:364) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:88) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:335) [elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Master Node Logs:
org.elasticsearch.snapshots.SnapshotException: [ultimate:athena/zADfqeXBSY6YnEWJu0FF4g] failed to read shard snapshot file for [main][2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1120) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:1000) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotsService.snapshotShards(SnapshotsService.java:603) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.action.admin.cluster.snapshots.status.TransportSnapshotsStatusAction.buildResponse(TransportSnapshotsStatusAction.java:235) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.action.admin.cluster.snapshots.status.TransportSnapshotsStatusAction.masterOperation(TransportSnapshotsStatusAction.java:101) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.action.admin.cluster.snapshots.status.TransportSnapshotsStatusAction.masterOperation(TransportSnapshotsStatusAction.java:63) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.0.jar:5.5.0]
Thank You