# Snapshots not working **URL:** \ **Category:** Elasticsearch\ **Created:** [August 23, 2018, 6:05am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653 "2018-08-23T06:05:56Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [August 23, 2018, 6:05am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/1 "2018-08-23T06:05:56Z")
Hi , I have installed and configured Elastic-search (Multi Node) Cluster on RHEL 6.9. Master : 192.168.2.79 Data-Node1 : 192.168.2.80 Data-Node2 : 192.168.2.81 I have set path.repo: ["/home/es-backup/es\_backup\_basic"] and permission for /home/es-backup/es\_backup\_basic 0777 in all 3 nodes. When I have executed the below command : [root@mcspm2 ~]# curl -X PUT "[http://192.168.2.79:9200/\_snapshot/my\_unverified\_backup?verify=false](http://192.168.2.79:9200/_snapshot/my_unverified_backup?verify=false)" -H 'Content-Type: application/json' -d'{"type": "fs","settings": {"location": "my\_unverified\_backup\_location"}}' {"acknowledged":true} root@mcspm2 ~]# curl -X POST "[http://192.168.2.79:9200/\_snapshot/my\_unverified\_backup/\_verify](http://192.168.2.79:9200/_snapshot/my_unverified_backup/_verify)" {"error":{"root\_cause":[{"type":"repository\_verification\_exception","reason":"[my\_unverified\_backup] [[A3w4wz7jSAG2nrzhEjLS0w, 'RemoteTransportException[[es-data2][192.168.2.81:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_unverified\_backup] a file written by master to the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] cannot be accessed on the node [{es-data2}{A3w4wz7jSAG2nrzhEjLS0w}{1iPxzhCOR8O77bc-x2mLpA}{192.168.2.81}{192.168.2.81:9300}]. This might indicate that the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [dwRRAbBcTEqJSQbG-bxc7A, 'RemoteTransportException[[es-data1][192.168.2.80:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_unverified\_backup] a file written by master to the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] cannot be accessed on the node [{es-data1}{dwRRAbBcTEqJSQbG-bxc7A}{rp1v9pBlTfWVl5h-6yiqCA}{192.168.2.80}{192.168.2.80:9300}]. This might indicate that the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"}],"type":"repository\_verification\_exception","reason":"[my\_unverified\_backup] [[A3w4wz7jSAG2nrzhEjLS0w, 'RemoteTransportException[[es-data2][192.168.2.81:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_unverified\_backup] a file written by master to the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] cannot be accessed on the node [{es-data2}{A3w4wz7jSAG2nrzhEjLS0w}{1iPxzhCOR8O77bc-x2mLpA}{192.168.2.81}{192.168.2.81:9300}]. This might indicate that the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [dwRRAbBcTEqJSQbG-bxc7A, 'RemoteTransportException[[es-data1][192.168.2.80:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_unverified\_backup] a file written by master to the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] cannot be accessed on the node [{es-data1}{dwRRAbBcTEqJSQbG-bxc7A}{rp1v9pBlTfWVl5h-6yiqCA}{192.168.2.80}{192.168.2.80:9300}]. This might indicate that the store [/home/es-backup/es\_backup\_basic/my\_unverified\_backup\_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"},"status":500} [root@mcspm2 ~]# Please provide us a solution. Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 23, 2018, 6:08am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/2 "2018-08-23T06:08:01Z")
The repository must be a shared file system accessible by all nodes in the cluster. The error message indicates that this is not the case. ---
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [August 23, 2018, 6:12am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/3 "2018-08-23T06:12:55Z")
Hi @Christian_Dahlqvist For cluster data backup, We must have a shared file system right ? Is there any other way to back my all cluster data and restore accordingly. Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 23, 2018, 6:14am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/4 "2018-08-23T06:14:03Z")
Yes, a shared file system is required. There is also plugins available that allows you to store snapshots in e.g. S3 or HDFS. ---
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [August 31, 2018, 8:18am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/5 "2018-08-31T08:18:01Z")
Hi @Christian_Dahlqvist Can shared file system be configured without SAN or NAS ? If yes, then how? Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 31, 2018, 8:45am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/6 "2018-08-31T08:45:54Z")
Not that I know of. ---
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [September 6, 2018, 10:03am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/7 "2018-09-06T10:03:48Z")
Hi @Christian_Dahlqvist, We have configured shared file system. We are running below command : GET /\_snapshot/\_all Response is : { "my\_backup": { "type": "fs", "settings": { "location": "/home/demo/" } }, "my\_fs\_backup": { "type": "fs", "settings": { "compress": "true", "location": "/home/demo/" } } } POST /\_snapshot/my\_fs\_backup/\_verify Response is : { "error": { "root\_cause": [ { "type": "repository\_verification\_exception", "reason": "[my\_fs\_backup] [[jQ15JZVdQoyOPMBMLDa8bQ, 'RemoteTransportException[[es-data1][192.168.2.80:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_fs\_backup] store location [/home/demo] is not accessible on the node [{es-data1}{jQ15JZVdQoyOPMBMLDa8bQ}{ZcJ5hVJtSEG1cgJORQ3kMQ}{192.168.2.80}{192.168.2.80:9300}{ml.machine\_memory=33654394880, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}]]; nested: AccessDeniedException[/home/demo/tests-zOCThU0ATX20BCZg5QonCw/data-jQ15JZVdQoyOPMBMLDa8bQ.dat];']]" } ], "type": "repository\_verification\_exception", "reason": "[my\_fs\_backup] [[jQ15JZVdQoyOPMBMLDa8bQ, 'RemoteTransportException[[es-data1][192.168.2.80:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_fs\_backup] store location [/home/demo] is not accessible on the node [{es-data1}{jQ15JZVdQoyOPMBMLDa8bQ}{ZcJ5hVJtSEG1cgJORQ3kMQ}{192.168.2.80}{192.168.2.80:9300}{ml.machine\_memory=33654394880, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}]]; nested: AccessDeniedException[/home/demo/tests-zOCThU0ATX20BCZg5QonCw/data-jQ15JZVdQoyOPMBMLDa8bQ.dat];']]" }, "status": 500 } can you tell us, why we are getting the above error. Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2018, 10:16am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/8 "2018-09-06T10:16:20Z")
I looks like the `/home/demo/` path is not read and writable from all nodes in the cluster. Is this really a shared file system? ---
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [September 6, 2018, 10:38am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/9 "2018-09-06T10:38:37Z")
Hi @Christian_Dahlqvist, I have created a directory 'test' under /home/demo and put a 'sample.txt' under /home/demo/test/ I can see the content of the /home/demo/test/sample.txt from every node. Master : 192.168.2.79 Data-Node1 : 192.168.2.80 Data-Node2 : 192.168.2.81 path.repo is set in all node's elasticsearch.yml file : /home/demo my storage is located at 192.168.2.82 under /home/elaback/ I can see sample.txt from all node's /home/demo/test/ and in 192.168.2.82 under /home/elaback/test/sample.txt This is our observation. Please correct me if anything is wrong. Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2018, 10:46am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/10 "2018-09-06T10:46:47Z")
Can all nodes also write to this directory? ---
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [September 6, 2018, 11:07am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/11 "2018-09-06T11:07:45Z")
Hi @Christian_Dahlqvist, yes, all nodes can write to this directory. I have modified sample.txt from each node one by one. And each modification reflect to all node's sample.txt. Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2018, 11:12am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/12 "2018-09-06T11:12:45Z")
Are you testing this as the user Elasticsearch runs under? ---
**Author:** ![amit.patra](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Post date:** [September 6, 2018, 11:18am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/13 "2018-09-06T11:18:06Z")
Hi @Christian_Dahlqvist, We have logged in as root user and modified sample.txt ls -ltr /home/demo/test/ -rw-r--r-- 1 nfsnobody nfsnobody 64 Sep 6 16:33 sample.txt Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2018, 11:22am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/14 "2018-09-06T11:22:15Z")
You need to be able to read and write from all nodes as the user Elasticsearch runs as, which cannot be `root`. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2018, 11:22am UTC](https://discuss.elastic.co/t/snapshots-not-working/145653/15 "2018-10-04T11:22:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.