# Unable to make snapshots to NFS filesystem **URL:** \ **Category:** Elasticsearch\ **Created:** [September 29, 2014, 8:45am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997 "2014-09-29T08:45:57Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Alex\_Harvey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_harvey/32/32183_2.png) [@Alex\_Harvey](https://discuss.elastic.co/u/Alex_Harvey)\ **Post date:** [September 29, 2014, 8:45am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997/1 "2014-09-29T08:45:57Z")
Hi all, I have been struggling to put together a backup solution for my ES cluster. As far as I understand the documentation at [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html) I can't understand why the following might be failing: I have exported an NFS filesystem to both nodes of my 2-node ES cluster, mounted as /srv/backup. I created the elastic search user on the NFS server too and then [root@back01 ~]# ls -ld /srv/backup/es\_backup drwxrwx---. 3 elasticsearch elasticsearch 4096 Sep 29 18:37 /srv/backup/es\_backup Start with a clean filesystem: [root@logdata01 ~]# rm -rf /srv/backup/\* Register the backup area: [root@logdata01 ~]# curl -s -XPUT [http://localhost:9200/\_snapshot/backup](http://localhost:9200/_snapshot/backup) -d '{ "type": "fs", "settings": { ``` "location": "/srv/backup" ``` } }' {"acknowledged":true} Create a snapshot: [root@logdata01 ~]# curl -XPUT "localhost:9200/\_snapshot/backup/tcom\_snapshot?wait\_for\_completion=true&pretty" I then get failures on various shards > Any help on how I could get this cluster into a sane state that can be backed up greatly appreciated. Best regards Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 29, 2014, 10:02am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997/2 "2014-09-29T10:02:15Z")
Can you do an ls -ld /srv/backup and provide the output? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 29 September 2014 18:45, Alex Harvey [alexharv074@gmail.com](mailto:alexharv074@gmail.com) wrote: > Hi all, > > I have been struggling to put together a backup solution for my ES cluster. > > As far as I understand the documentation at > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html) > > I can't understand why the following might be failing: > > I have exported an NFS filesystem to both nodes of my 2-node ES cluster, > mounted as /srv/backup. > > I created the Elasticsearch user on the NFS server too and then > > [root@back01 ~]# ls -ld /srv/backup/es\_backup > > drwxrwx---. 3 elasticsearch elasticsearch 4096 Sep 29 18:37 > /srv/backup/es\_backup > > Start with a clean filesystem: > > [root@logdata01 ~]# rm -rf /srv/backup/\* > > Register the backup area: > > [root@logdata01 ~]# curl -s -XPUT [http://localhost:9200/\_snapshot/backup](http://localhost:9200/_snapshot/backup) > -d '{ > > "type": "fs", > > "settings": { > > ``` > "location": "/srv/backup" > > ``` > > } > > }' > > {"acknowledged":true} > > Create a snapshot: > > [root@logdata01 ~]# curl -XPUT > "localhost:9200/\_snapshot/backup/tcom\_snapshot?wait\_for\_completion=true&pretty" > > I then get failures on various shards > [gist:b4c7d35028c425f70f20 · GitHub](https://gist.github.com/alexharv074/b4c7d35028c425f70f20) > > Any help on how I could get this cluster into a sane state that can be > backed up greatly appreciated. > > Best regards > Alex > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bCJ252AMkE434HwkB7xZVFuMxvxZQK6hZdJCgquQFdig%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bCJ252AMkE434HwkB7xZVFuMxvxZQK6hZdJCgquQFdig%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alex\_Harvey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_harvey/32/32183_2.png) [@Alex\_Harvey](https://discuss.elastic.co/u/Alex_Harvey)\ **Post date:** [September 30, 2014, 12:30am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997/3 "2014-09-30T00:30:08Z")
Thanks for responding. It doesn't seem to be a permissions problem - [root@logdata01 ~]# ls -ld /srv/backup drwxrwx---. 3 elasticsearch elasticsearch 4096 Sep 29 18:43 /srv/backup [root@logdata01 ~]# find /srv/backup/ ! -user elasticsearch -or ! -group elasticsearch [root@logdata01 ~]# [root@logdata01 ~]# find /srv/backup -ls |head 131073 4 drwxrwx--- 3 elasticsearch elasticsearch 4096 Sep 29 18:43 /srv/backup 131076 4 drwxr-xr-x 12 elasticsearch elasticsearch 4096 Sep 29 18:37 /srv/backup/indices 131095 4 drwxr-xr-x 6 elasticsearch elasticsearch 4096 Sep 29 18:42 /srv/backup/indices/logstash-2014.09.28 131096 8 -rw-r--r-- 1 elasticsearch elasticsearch 4120 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/snapshot-tcom\_snapshot 131189 4 drwxr-xr-x 2 elasticsearch elasticsearch 4096 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/3 131193 8 -rw-r--r-- 1 elasticsearch elasticsearch 4443 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_3 131201 4 -rw-r--r-- 1 elasticsearch elasticsearch 689 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_b 131202 4 -rw-r--r-- 1 elasticsearch elasticsearch 61 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_c 131206 4 -rw-r--r-- 1 elasticsearch elasticsearch 281 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_g 131200 4 -rw-r--r-- 1 elasticsearch elasticsearch 349 Sep 29 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_a On Monday, September 29, 2014 8:02:42 PM UTC+10, Mark Walkom wrote: > Can you do an ls -ld /srv/backup and provide the output? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 29 September 2014 18:45, Alex Harvey \<[alexh...@gmail.com](mailto:alexh...@gmail.com) \\> > wrote: > > > Hi all, > > > > I have been struggling to put together a backup solution for my ES > > cluster. > > > > As far as I understand the documentation at > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html) > > > > I can't understand why the following might be failing: > > > > I have exported an NFS filesystem to both nodes of my 2-node ES cluster, > > mounted as /srv/backup. > > > > I created the Elasticsearch user on the NFS server too and then > > > > [root@back01 ~]# ls -ld /srv/backup/es\_backup > > > > drwxrwx---. 3 elasticsearch elasticsearch 4096 Sep 29 18:37 > > /srv/backup/es\_backup > > > > Start with a clean filesystem: > > > > [root@logdata01 ~]# rm -rf /srv/backup/\* > > > > Register the backup area: > > > > [root@logdata01 ~]# curl -s -XPUT [http://localhost:9200/\_snapshot/backup](http://localhost:9200/_snapshot/backup) > > -d '{ > > > > "type": "fs", > > > > "settings": { > > > > ``` > > "location": "/srv/backup" > > > > ``` > > > > } > > > > }' > > > > {"acknowledged":true} > > > > Create a snapshot: > > > > [root@logdata01 ~]# curl -XPUT > > "localhost:9200/\_snapshot/backup/tcom\_snapshot?wait\_for\_completion=true&pretty" > > > > I then get failures on various shards > > [gist:b4c7d35028c425f70f20 · GitHub](https://gist.github.com/alexharv074/b4c7d35028c425f70f20) > > > > Any help on how I could get this cluster into a sane state that can be > > backed up greatly appreciated. > > > > Best regards > > Alex > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1160a631-29b6-4242-97ec-67f446da81bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1160a631-29b6-4242-97ec-67f446da81bb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![hakman](https://avatars.discourse-cdn.com/v4/letter/h/59ef9b/32.png) [@hakman](https://discuss.elastic.co/u/hakman)\ **Post date:** [September 30, 2014, 4:36am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997/4 "2014-09-30T04:36:19Z")
Hi Alex, Any chance you have disk quota enabled for the NFS share? I see this is the snapshot output: > "IndexShardSnapshotFailedException[[logstash-2014.09.19][4] Failed to > perform snapshot (index files)]; nested: IOException[No space left on > device]; " Can you try copying a larger file to the NFS server as user elasticsearch? ## Regards, Ciprian Hacman Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Tuesday, September 30, 2014 3:30:08 AM UTC+3, Alex Harvey wrote: > Thanks for responding. > > It doesn't seem to be a permissions problem - > > [root@logdata01 ~]# ls -ld /srv/backup > > drwxrwx---. 3 elasticsearch elasticsearch 4096 Sep 29 18:43 /srv/backup > > [root@logdata01 ~]# find /srv/backup/ ! -user elasticsearch -or ! -group > elasticsearch > > [root@logdata01 ~]# > > [root@logdata01 ~]# find /srv/backup -ls |head > > 131073 4 drwxrwx--- 3 elasticsearch elasticsearch 4096 Sep 29 > 18:43 /srv/backup > > 131076 4 drwxr-xr-x 12 elasticsearch elasticsearch 4096 Sep 29 > 18:37 /srv/backup/indices > > 131095 4 drwxr-xr-x 6 elasticsearch elasticsearch 4096 Sep 29 > 18:42 /srv/backup/indices/logstash-2014.09.28 > > 131096 8 -rw-r--r-- 1 elasticsearch elasticsearch 4120 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/snapshot-tcom\_snapshot > > 131189 4 drwxr-xr-x 2 elasticsearch elasticsearch 4096 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/3 > > 131193 8 -rw-r--r-- 1 elasticsearch elasticsearch 4443 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_3 > > 131201 4 -rw-r--r-- 1 elasticsearch elasticsearch 689 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_b > > 131202 4 -rw-r--r-- 1 elasticsearch elasticsearch 61 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_c > > 131206 4 -rw-r--r-- 1 elasticsearch elasticsearch 281 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_g > > 131200 4 -rw-r--r-- 1 elasticsearch elasticsearch 349 Sep 29 > 18:37 /srv/backup/indices/logstash-2014.09.28/3/\_\_a > > On Monday, September 29, 2014 8:02:42 PM UTC+10, Mark Walkom wrote: > > > Can you do an ls -ld /srv/backup and provide the output? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 29 September 2014 18:45, Alex Harvey [alexh...@gmail.com](mailto:alexh...@gmail.com) wrote: > > > > > Hi all, > > > > > > I have been struggling to put together a backup solution for my ES > > > cluster. > > > > > > As far as I understand the documentation at > > > > > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html) > > > > > > I can't understand why the following might be failing: > > > > > > I have exported an NFS filesystem to both nodes of my 2-node ES cluster, > > > mounted as /srv/backup. > > > > > > I created the Elasticsearch user on the NFS server too and then > > > > > > [root@back01 ~]# ls -ld /srv/backup/es\_backup > > > > > > drwxrwx---. 3 elasticsearch elasticsearch 4096 Sep 29 18:37 > > > /srv/backup/es\_backup > > > > > > Start with a clean filesystem: > > > > > > [root@logdata01 ~]# rm -rf /srv/backup/\* > > > > > > Register the backup area: > > > > > > [root@logdata01 ~]# curl -s -XPUT [http://localhost:9200/\_snapshot/backup](http://localhost:9200/_snapshot/backup) > > > -d '{ > > > > > > "type": "fs", > > > > > > "settings": { > > > > > > ``` > > > "location": "/srv/backup" > > > > > > ``` > > > > > > } > > > > > > }' > > > > > > {"acknowledged":true} > > > > > > Create a snapshot: > > > > > > [root@logdata01 ~]# curl -XPUT > > > "localhost:9200/\_snapshot/backup/tcom\_snapshot?wait\_for\_completion=true&pretty" > > > > > > I then get failures on various shards > > > [https://gist.github.com/alexharv074/b4c7d35028c425f70f20](https://gist.github.com/alexharv074/b4c7d35028c425f70f20) > > > > > > Any help on how I could get this cluster into a sane state that can be > > > backed up greatly appreciated. > > > > > > Best regards > > > Alex > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ec09f9bd-5075-4bd5-adf7-d88cbb636c1f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c968bd02-6dd4-4c38-ae59-0b2f69b1ea32%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c968bd02-6dd4-4c38-ae59-0b2f69b1ea32%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alex\_Harvey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_harvey/32/32183_2.png) [@Alex\_Harvey](https://discuss.elastic.co/u/Alex_Harvey)\ **Post date:** [October 8, 2014, 7:05am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997/5 "2014-10-08T07:05:29Z")
Ciprian, Thanks for your input - I had indeed missed that disk space failure and it turns out I was hitting an intermittent disk space issue. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02b41de5-478e-4083-b5d2-c9b493f24732%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02b41de5-478e-4083-b5d2-c9b493f24732%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/unable-to-make-snapshots-to-nfs-filesystem/19997/6 "2017-07-06T00:57:45Z")