# Snapshot repository missing after remounting network disk **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [May 17, 2022, 11:01am UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921 "2022-05-17T11:01:57Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 17, 2022, 11:01am UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/1 "2022-05-17T11:01:57Z")
Hi All, We decided to increase the amount of storage we have for snapshots On a three node cluster elastic cluster v7. We did that by copying the contents of the old disk to the bigger disk. When copying was done, we unmounted the old disk and mounted the new one while the cluster was running. The nfs unmounted the disk, so we had to remount it. We have the shared disk on one of the nodes and share the repository using nfs. Now, when I request ```auto GET /_cat/snapshots ``` I get the following error ```auto { "error" : { "root_cause" : [ { "type" : "action_request_validation_exception", "reason" : "Validation Failed: 1: repository is missing;" } ], "type" : "action_request_validation_exception", "reason" : "Validation Failed: 1: repository is missing;" }, "status" : 400 } ``` I can now access the storage on the three servers just fine. However, I did not restart the servers yet. Any idea how to solve this? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 17, 2022, 12:21pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/2 "2022-05-17T12:21:32Z")
> [@elaskibuser](#): > > I did not restart the servers yet. You need to restart every node, since you have three nodes, you can do a rolling restart to avoid downtime of the cluster. ---
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 17, 2022, 6:44pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/3 "2022-05-17T18:44:14Z")
Hi Thanks for your reply! I restarted the cluster. However `GET /_cat/snapshots` still gives the same response ```auto { "error" : { "root_cause" : [ { "type" : "action_request_validation_exception", "reason" : "Validation Failed: 1: repository is missing;" } ], "type" : "action_request_validation_exception", "reason" : "Validation Failed: 1: repository is missing;" }, "status" : 400 } ``` However, now, getting the snapshots as follows `GET /_snapshot/repo_name/snap_name` works fine ---
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 18, 2022, 2:34pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/4 "2022-05-18T14:34:37Z")
Is it possible that the `GET /_cat/snapshots` is only introduced after Elasticsearch v7, although it is complaining about a repository. EDIT: No, a snapshot failed yesterday citing an internal error. I already tried a rolling cluster restart and full restart. here is the repo part in Elasticsearch.yml file ```auto path.repo: ["/esdata/nfs/elasticsearch/backups"] ``` When I inspect the docker container for mounts, here is the path Node 1: ```auto "Type": "bind", "Source": "/esdata/nfs/elasticsearch", "Destination": "/esdata/nfs/elasticsearch", "Mode": "", "RW": true, "Propagation": "rprivate" ``` Node 2: ```auto "Type": "bind", "Source": "/esdata/nfs/elasticsearch", "Destination": "/esdata/nfs/elasticsearch", "Mode": "", "RW": true, "Propagation": "rprivate" ``` Node 3: ```auto "Type": "bind", "Source": "/esdata/nfs/elasticsearch", "Destination": "/esdata/nfs/elasticsearch", "Mode": "", "RW": true, "Propagation": "rprivate" ``` When I verify my repositories, using the verify API `POST /_snapshot/my_repository/_verify`, it works fine. All nodes show up. I can also create repositories. ---
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 18, 2022, 3:29pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/5 "2022-05-18T15:29:09Z")
I just checked the snapshot status, it seems the failure was due to the shutdown ```auto { "index" : "index-2022.05.17", "index_uuid" : "index-2022.05.17", "shard_id" : 3, "reason" : "node shutdown", "node_id" : "---------------------------", "status" : "INTERNAL_SERVER_ERROR" } ``` However, the repository error is still unexplained? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 18, 2022, 7:02pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/6 "2022-05-18T19:02:19Z")
The endpoint `GET _cat/snapshots` exists since version 6.X at least. What do you have in Elasticsearch logs? Is the snapshot registered and mounted in all three nodes? ---
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 18, 2022, 9:49pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/7 "2022-05-18T21:49:56Z")
Hi, Thanks for the reply! How can I make sure that the snapshot is registered and mounted on all nodes? I checked through Kibana that all repositories are registered. I even deleted the repositories and registered them again as shared location. it was able to find all snapshots. Repository verification also works for all repositories. I also tried to delete/create/ and restore snapshots. In Elastic logs, I found the following error: ```auto [2022-05-17T12:16:47,077][ERROR][o.e.x.s.SnapshotLifecycleTask] [node2]failed to create snapshot for snapshot lifecycle policy [index-daily-snapshot]: SnapshotException[[backup-v7-index:index-2022.05.16------/-----------] failed to update snapshot in repository]; nested: ElasticsearchException[failed to create blob container]; nested: FileSystemException[/esdata/nfs/elasticsearch/backups/v7/index/live/indices: Stale file handle]; uncaught exception in thread [main] java.lang.IllegalStateException: Unable to access 'path.repo' (/esdata/nfs/elasticsearch/backups) Likely root cause: java.nio.file.AccessDeniedException: /esdata/nfs/elasticsearch/backups at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) For complete error details, refer to the log at /usr/share/elasticsearch/logs/es-cluster.log uncaught exception in thread [main] ``` however, when I check the access rights it is as follows: for /esdata/nfs: ```auto drwxr-xr-x 6 user user 4096 May 17 12:58 nfs ``` for /esdata/nfs/Elasticsearch: ```auto drwxrwxr-x 3 adminuser adminuser 4096 Aug 10 2020 elasticsearch ``` for /esdata/nfs/Elasticsearch/backups: ```auto drwxrwxr-x 7 adminuser adminuser 4096 Jul 13 2021 backups ``` I cannot find the logs mentioned (/usr/share/Elasticsearch/logs/es-cluster.log) as it seem is not written. However, I believe those errors are back from when I was restarting the nodes. I am finding it hard to believe it is an access issue as elastic can write and read the snaps. ---
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 20, 2022, 12:05pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/8 "2022-05-20T12:05:22Z")
Also, SLM does not seem to be complaining, when I run GET \_slm/status, it returns as RUNNING just a friendly reminder as I still have this issue. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 20, 2022, 12:33pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/9 "2022-05-20T12:33:26Z")
Never saw this issue and I do not run docker, so I'm not sure if it is related to docker or note, but this log line could give some hint: ```auto FileSystemException[/esdata/nfs/elasticsearch/backups/v7/index/live/indices: Stale file handle ``` Since you unmounted and mounted the repository while the cluster was running, this could have caused some issue. Do you had some running snapshots that maybe were running while the mounting/umounting happened? The user running Elasticsearch is the same that owns the paths? You will need to see if someone from Elastic can give more context about what this means, but just a remind that there is no SLA in this forum. ---
**Author:** ![elaskibuser](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elaskibuser](https://discuss.elastic.co/u/elaskibuser)\ **Post date:** [May 20, 2022, 12:50pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/10 "2022-05-20T12:50:45Z")
Thanks a ton for the reply. It is very likely that there was a running snapshot when mounting/unmounting happened unfortunately. Yes it is the same user I understand that and I appreciate your help! Thanks again! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 17, 2022, 12:51pm UTC](https://discuss.elastic.co/t/snapshot-repository-missing-after-remounting-network-disk/304921/11 "2022-06-17T12:51:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.