# Snapshots stopped working after switching from a single server to cluster (getting the server married?) **URL:** \ **Category:** Elasticsearch\ **Created:** [July 27, 2019, 6:39pm UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531 "2019-07-27T18:39:01Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 27, 2019, 6:39pm UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/1 "2019-07-27T18:39:02Z")
Simple as that. I'm attempting to take snapshots from the same server as before, which isn't the master anymore. path.repo: ["/var/db/elasticsearch"] Snapshot directory is /var/db/elasticsearch/backup. For some reason its index-NNN files stopped updating on the day of the switch, judging by FS file last modification time. But other data files seem to be updated. I should note that on other two servers of the cluster backup directory also exists, but has other index-NNN files which started from 1 and going up every day by 1. Why is the backup directory part of the cluster at all? Would moving it outside of repo.path fix the problem? ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 27, 2019, 6:44pm UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/2 "2019-07-27T18:44:31Z")
I should note that even though I still keep addressing the previous master (now slave) in my API requests (using curl), the log entries regarding backup snapshot creation etc. magically appear in the master log files [2019-07-27T02:31:00,454][INFO][o.e.s.SnapshotsService] [[myserver.com](http://myserver.com)] snapshot [backup:snapshot/W3FHurPGSa6MkxFDcRHDKQ] deleted [2019-07-27T02:31:00,466][INFO][o.e.r.RepositoriesService] [[myserver.com](http://myserver.com)] delete repository [backup] [2019-07-27T02:31:00,493][INFO][o.e.r.RepositoriesService] [[myserver.com](http://myserver.com)] put repository [backup] [2019-07-27T02:31:00,534][INFO][o.e.s.SnapshotsService] [[myserver.com](http://myserver.com)] snapshot [backup:snapshot/Vmr9WpUDS3CeRAXRkAJw8A] started [2019-07-27T02:34:29,411][INFO][o.e.s.SnapshotsService] [[myserver.com](http://myserver.com)] snapshot [backup:snapshot/Vmr9WpUDS3CeRAXRkAJw8A] completed with state [SUCCESS] but on both slaves (is this the correct term?) I'm getting this instead: [2019-07-27T02:31:00,515][WARN][o.e.r.VerifyNodeRepositoryAction] [[myslave.com](http://myslave.com)] [backup] failed to verify repository org.elasticsearch.repositories.RepositoryVerificationException: [backup] a file written by master to the store [/var/db/elasticsearch/backup] cannot be accessed on the node [{[myslave.com](http://myslave.com)}{\_7zxUfdrQrSDyollCyytSg}{bN\_ptp5jS2mea1SFYcRs9w}{172.16.1.18}{172.16.1.18:9300}{xpack.installed=true}]. This might indicate that the store [/var/db/elasticsearch/backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 27, 2019, 6:50pm UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/3 "2019-07-27T18:50:57Z")
Damn. [Here](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html) it says that a shared filesystem needs to be used for snapshots to work. What is it for? Any way to take snapshots without setting it up? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 27, 2019, 8:08pm UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/4 "2019-07-27T20:08:21Z")
No, the snapshot is created cluster wide so ALL nodes need to have access to the same repository. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 27, 2019, 9:27pm UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/5 "2019-07-27T21:27:42Z")
> [@rihad](#): > > slaves (is this the correct term?) No it's really not. They're either data nodes (if `node.master: false` and `node.data: true`) or just "the other nodes". > [@rihad](#): > > Any way to take snapshots without setting it up? You could try setting up a [Minio](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-client.html#repository-s3-compatible-services) node to emulate AWS S3, so that you can use the `repository-s3` plugin. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 28, 2019, 5:24am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/6 "2019-07-28T05:24:27Z")
Is there a simpler way? Can I set up the backup with no sharding, but only with replication? All I need is a snapshot that I can copy to an entirely other machine (staging environment) daily and restore it there. This worked when the "cluster" consisted of a single node. At the very least, would taking the backup on the master only work? ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 28, 2019, 6:59am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/7 "2019-07-28T06:59:08Z")
How can the NFS server be set up? Should all data nodes be given read+write access to the shared directory? ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 28, 2019, 8:12am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/8 "2019-07-28T08:12:22Z")
Awesome! The nfs setup worked fine, and I could still target any server belonging to the cluster when issuing the queries, not only the master (maybe it had to be a master-eligible one, but this is irrelevant). Thanks to everyone ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 28, 2019, 8:25am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/9 "2019-07-28T08:25:20Z")
> [@rihad](#): > > would taking the backup on the master only work? No, as a rule the master does its best to avoid handling any data, but this idea would route the whole snapshot through the master. It's the primary of each shard that takes the snapshot of that shard. > [@rihad](#): > > I could still target any server belonging to the cluster when issuing the queries, not only the master Yes, you can send any request to any node in a cluster and rely on Elasticsearch to make sure it gets to the right place. In larger clusters it's often a good idea to avoid sending any requests directly to the master nodes so that they can focus on coordinating the cluster. Glad you got it working, thanks for letting us know. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 28, 2019, 10:11am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/10 "2019-07-28T10:11:52Z")
Thanks. I'm a bit wary of the fact that three processes on different machines have full r/w access to the same shared data directory (/var/db/elasticsearch/backup). Will they make sure not to step on each other's toes? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 28, 2019, 10:56am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/11 "2019-07-28T10:56:44Z")
Yes, the master makes sure that they all stay out of each others' way when making a snapshot. You can't have more than one _cluster_ writing to the same snapshot repository, because there's no way to coordinate that, but it's fine if it's just a single cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 25, 2019, 10:56am UTC](https://discuss.elastic.co/t/snapshots-stopped-working-after-switching-from-a-single-server-to-cluster-getting-the-server-married/192531/12 "2019-08-25T10:56:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.