# Why does creating a repository fail? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 16, 2015, 8:33pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697 "2015-03-16T20:33:10Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 16, 2015, 8:33pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/1 "2015-03-16T20:33:10Z")
Why does this happen? curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > ``` > > "type": "fs", > > "settings": { > > "location": "/mounts/prod_backup/my_backup", > > "compress": true > > } > > > > ``` > > > > }' > > { > > "error" : > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > nested: RepositoryVerificationException[[my\_backup] > > [vxUQwUTCQwOaLyCy0eMK8A, > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > nested: RepositoryVerificationException[[my\_backup] store location > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > nested: > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > nested: RepositoryVerificationException[[my\_backup] store location > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > nested: > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > nested: RepositoryVerificationException[[my\_backup] store location > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > nested: > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > (Permission denied)]; ']]]; ", > > "status" : 500 > > } /mounts/prod\_backup is an NFS share mounted on all my ES nodes. The only thing I can think of is that file permissions are wrong. But I'm not sure how to fix that. On the two nodes not listed in the error, the elasticsearch gid and uid are different that on the other three nodes. So I can't just _chown -R elasticsearch:elasticsearch_. Do I have to make sure that the uid and gid are the same on all nodes? How am I supposed to do that when I install from the apt repositories? Also, log-estools-01 has _data.node = false_. Why does it still need the prod\_backup mount? log-estools-01 is on Ubuntu 14.04. Everything else is Ubuntu 12.04. Running ES 1.4.4. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 16, 2015, 10:47pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/2 "2015-03-16T22:47:35Z")
Can you post the output from _mount_ and _ls -ld /mounts /mounts/prod\_backup_? On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > Why does this happen? > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > > > ``` > > > "type": "fs", > > > "settings": { > > > "location": "/mounts/prod_backup/my_backup", > > > "compress": true > > > } > > > > > > ``` > > > > > > }' > > > { > > > "error" : > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > nested: RepositoryVerificationException[[my\_backup] > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > nested: RepositoryVerificationException[[my\_backup] store location > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > nested: > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > nested: RepositoryVerificationException[[my\_backup] store location > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > nested: > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > nested: RepositoryVerificationException[[my\_backup] store location > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > nested: > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > (Permission denied)]; ']]]; ", > > > "status" : 500 > > > } > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > The only thing I can think of is that file permissions are wrong. But I'm > not sure how to fix that. > > On the two nodes not listed in the error, the elasticsearch gid and uid > are different that on the other three nodes. So I can't just _chown -R > elasticsearch:elasticsearch_. > > Do I have to make sure that the uid and gid are the same on all nodes? How > am I supposed to do that when I install from the apt repositories? > > Also, log-estools-01 has _data.node = false_. Why does it still need the > prod\_backup mount? > > log-estools-01 is on Ubuntu 14.04. > > Everything else is Ubuntu 12.04. > > Running ES 1.4.4. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 16, 2015, 11:04pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/3 "2015-03-16T23:04:56Z")
First, it is a file permissions issue. I did get snapshots to run when I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is 777. Prior to that it was 775 or 755. So, I could revise my question to "How can I get snapshots working without using insecure file permissions?" root@log-elasticsearch-01:~# mount /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) udev on /dev type devtmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev) /dev/sda1 on /boot type ext2 (rw) rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) root@log-elasticsearch-01:~# ls -ld /mounts drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 /mounts/prod\_backup/ --David Reagan On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > Can you post the output from _mount_ and _ls -ld /mounts > /mounts/prod\_backup_? > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > Why does this happen? > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > > > > > ``` > > > > "type": "fs", > > > > "settings": { > > > > "location": "/mounts/prod_backup/my_backup", > > > > "compress": true > > > > } > > > > > > > > ``` > > > > > > > > }' > > > > { > > > > "error" : > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > nested: RepositoryVerificationException[[my\_backup] > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > nested: > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > nested: > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > nested: > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > (Permission denied)]; ']]]; ", > > > > "status" : 500 > > > > } > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > The only thing I can think of is that file permissions are wrong. But I'm > > not sure how to fix that. > > > > On the two nodes not listed in the error, the elasticsearch gid and uid > > are different that on the other three nodes. So I can't just _chown -R > > elasticsearch:elasticsearch_. > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > How am I supposed to do that when I install from the apt repositories? > > > > Also, log-estools-01 has _data.node = false_. Why does it still need the > > prod\_backup mount? > > > > log-estools-01 is on Ubuntu 14.04. > > > > Everything else is Ubuntu 12.04. > > > > Running ES 1.4.4. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrew](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@andrew](https://discuss.elastic.co/u/andrew)\ **Post date:** [March 16, 2015, 11:40pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/4 "2015-03-16T23:40:15Z")
Hi David, This is a common problem with NFS. Unfortunately the protocol assumes identical uid/gid mappings across all machines. It’s just one of those annoying sys-admin tasks that one has to take into account when using NFS. To get your permissions back to less permissive settings you will have to edit the /etc/passwd and /etc/group files to keep them in sync. See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) for more context. - Andrew > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > First, it is a file permissions issue. I did get snapshots to run when I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is 777. Prior to that it was 775 or 755. So, I could revise my question to "How can I get snapshots working without using insecure file permissions?" > > root@log-elasticsearch-01:~# mount > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > proc on /proc type proc (rw,noexec,nosuid,nodev) > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > none on /sys/fs/fuse/connections type fusectl (rw) > none on /sys/kernel/debug type debugfs (rw) > none on /sys/kernel/security type securityfs (rw) > udev on /dev type devtmpfs (rw,mode=0755) > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > none on /run/shm type tmpfs (rw,nosuid,nodev) > /dev/sda1 on /boot type ext2 (rw) > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > root@log-elasticsearch-01:~# ls -ld /mounts > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 /mounts/prod\_backup/ > > --David Reagan > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom \<[markwalkom@gmail.com](mailto:markwalkom@gmail.com) [mailto:markwalkom@gmail.com](mailto:markwalkom@gmail.com)\> wrote: > Can you post the output from mount and ls -ld /mounts /mounts/prod\_backup? > > On 16 March 2015 at 13:33, David Reagan \<[jerrac@gmail.com](mailto:jerrac@gmail.com) [mailto:jerrac@gmail.com](mailto:jerrac@gmail.com)\> wrote: > Why does this happen? > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true) [http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > > ``` > > "type": "fs", > > "settings": { > > "location": "/mounts/prod_backup/my_backup", > > "compress": true > > } > > > > ``` > > > > }' > > { > > "error" : "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[my\_backup] [vxUQwUTCQwOaLyCy0eMK8A, 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_backup] store location [/mounts/prod\_backup/my\_backup] is not accessible on the node [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; nested: FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_backup] store location [/mounts/prod\_backup/my\_backup] is not accessible on the node [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; nested: FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_backup] store location [/mounts/prod\_backup/my\_backup] is not accessible on the node [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; nested: FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw (Permission denied)]; ']]]; ", > > "status" : 500 > > } > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > The only thing I can think of is that file permissions are wrong. But I'm not sure how to fix that. > > On the two nodes not listed in the error, the elasticsearch gid and uid are different that on the other three nodes. So I can't just chown -R elasticsearch:elasticsearch. > > Do I have to make sure that the uid and gid are the same on all nodes? How am I supposed to do that when I install from the apt repositories? > > Also, log-estools-01 has data.node = false. Why does it still need the prod\_backup mount? > > log-estools-01 is on Ubuntu 14.04. > > Everything else is Ubuntu 12.04. > > Running ES 1.4.4. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 16, 2015, 11:45pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/5 "2015-03-16T23:45:44Z")
You can do a uid remap for NFS mounts, eg [http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/](http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/) On 16 March 2015 at 16:04, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > First, it is a file permissions issue. I did get snapshots to run when I > chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > 777. Prior to that it was 775 or 755. So, I could revise my question to > "How can I get snapshots working without using insecure file permissions?" > > root@log-elasticsearch-01:~# mount > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > proc on /proc type proc (rw,noexec,nosuid,nodev) > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > none on /sys/fs/fuse/connections type fusectl (rw) > none on /sys/kernel/debug type debugfs (rw) > none on /sys/kernel/security type securityfs (rw) > udev on /dev type devtmpfs (rw,mode=0755) > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > none on /run/shm type tmpfs (rw,nosuid,nodev) > /dev/sda1 on /boot type ext2 (rw) > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type > nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > root@log-elasticsearch-01:~# ls -ld /mounts > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > /mounts/prod\_backup/ > > --David Reagan > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > > > Can you post the output from _mount_ and _ls -ld /mounts > > /mounts/prod\_backup_? > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > Why does this happen? > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > > > > > > > ``` > > > > > "type": "fs", > > > > > "settings": { > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > "compress": true > > > > > } > > > > > > > > > > ``` > > > > > > > > > > }' > > > > > { > > > > > "error" : > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > nested: > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > nested: > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > nested: > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > (Permission denied)]; ']]]; ", > > > > > "status" : 500 > > > > > } > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > The only thing I can think of is that file permissions are wrong. But > > > I'm not sure how to fix that. > > > > > > On the two nodes not listed in the error, the elasticsearch gid and uid > > > are different that on the other three nodes. So I can't just _chown -R > > > elasticsearch:elasticsearch_. > > > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > > How am I supposed to do that when I install from the apt repositories? > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > the prod\_backup mount? > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > Everything else is Ubuntu 12.04. > > > > > > Running ES 1.4.4. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_XqNm2S91VEMKCHpoxWzvxxLzHPBhC%3D66nNbSLhY%2BaoA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_XqNm2S91VEMKCHpoxWzvxxLzHPBhC%3D66nNbSLhY%2BaoA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 16, 2015, 11:46pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/6 "2015-03-16T23:46:31Z")
Would an iSCSI mount have the same issue? I believe our SAN supports both. --David Reagan On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > Hi David, > > This is a common problem with NFS. Unfortunately the protocol assumes > identical uid/gid mappings across all machines. It’s just one of those > annoying sys-admin tasks that one has to take into account when using NFS. > To get your permissions back to less permissive settings you will have to > edit the /etc/passwd and /etc/group files to keep them in sync. > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) for > more context. > > - Andrew > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > First, it is a file permissions issue. I did get snapshots to run when I > chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > 777. Prior to that it was 775 or 755. So, I could revise my question to > "How can I get snapshots working without using insecure file permissions?" > > root@log-elasticsearch-01:~# mount > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > proc on /proc type proc (rw,noexec,nosuid,nodev) > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > none on /sys/fs/fuse/connections type fusectl (rw) > none on /sys/kernel/debug type debugfs (rw) > none on /sys/kernel/security type securityfs (rw) > udev on /dev type devtmpfs (rw,mode=0755) > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > none on /run/shm type tmpfs (rw,nosuid,nodev) > /dev/sda1 on /boot type ext2 (rw) > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type > nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > root@log-elasticsearch-01:~# ls -ld /mounts > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > /mounts/prod\_backup/ > > --David Reagan > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > > > Can you post the output from _mount_ and _ls -ld /mounts > > /mounts/prod\_backup_? > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > Why does this happen? > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > > > > > > > ``` > > > > > "type": "fs", > > > > > "settings": { > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > "compress": true > > > > > } > > > > > > > > > > ``` > > > > > > > > > > }' > > > > > { > > > > > "error" : > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > nested: > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > nested: > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > nested: > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > (Permission denied)]; ']]]; ", > > > > > "status" : 500 > > > > > } > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > The only thing I can think of is that file permissions are wrong. But > > > I'm not sure how to fix that. > > > > > > On the two nodes not listed in the error, the elasticsearch gid and uid > > > are different that on the other three nodes. So I can't just _chown -R > > > elasticsearch:elasticsearch_. > > > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > > How am I supposed to do that when I install from the apt repositories? > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > the prod\_backup mount? > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > Everything else is Ubuntu 12.04. > > > > > > Running ES 1.4.4. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrew](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@andrew](https://discuss.elastic.co/u/andrew)\ **Post date:** [March 17, 2015, 12:20am UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/7 "2015-03-17T00:20:34Z")
I’m not that familiar with iSCSI so I hesitate to say for sure, but anytime you are cross-mounting filesystems on Linux you have to take uid/gid consistency into account. - Andrew > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > Would an iSCSI mount have the same issue? I believe our SAN supports both. > > --David Reagan > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden \<[andrew@elastic.co](mailto:andrew@elastic.co) [mailto:andrew@elastic.co](mailto:andrew@elastic.co)\> wrote: > Hi David, > > This is a common problem with NFS. Unfortunately the protocol assumes identical uid/gid mappings across all machines. It’s just one of those annoying sys-admin tasks that one has to take into account when using NFS. To get your permissions back to less permissive settings you will have to edit the /etc/passwd and /etc/group files to keep them in sync. > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) for more context. > > - Andrew > > > On Mar 16, 2015, at 4:04 PM, David Reagan \<[jerrac@gmail.com](mailto:jerrac@gmail.com) [mailto:jerrac@gmail.com](mailto:jerrac@gmail.com)\> wrote: > > > > First, it is a file permissions issue. I did get snapshots to run when I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is 777. Prior to that it was 775 or 755. So, I could revise my question to "How can I get snapshots working without using insecure file permissions?" > > > > root@log-elasticsearch-01:~# mount > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > none on /sys/fs/fuse/connections type fusectl (rw) > > none on /sys/kernel/debug type debugfs (rw) > > none on /sys/kernel/security type securityfs (rw) > > udev on /dev type devtmpfs (rw,mode=0755) > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > /dev/sda1 on /boot type ext2 (rw) > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 /mounts/prod\_backup/ > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom \<[markwalkom@gmail.com](mailto:markwalkom@gmail.com) [mailto:markwalkom@gmail.com](mailto:markwalkom@gmail.com)\> wrote: > > Can you post the output from mount and ls -ld /mounts /mounts/prod\_backup? > > > > On 16 March 2015 at 13:33, David Reagan \<[jerrac@gmail.com](mailto:jerrac@gmail.com) [mailto:jerrac@gmail.com](mailto:jerrac@gmail.com)\> wrote: > > Why does this happen? > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true) [http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d '{ > > > > > ``` > > > "type": "fs", > > > "settings": { > > > "location": "/mounts/prod_backup/my_backup", > > > "compress": true > > > } > > > > > > ``` > > > > > > }' > > > { > > > "error" : "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[my\_backup] [vxUQwUTCQwOaLyCy0eMK8A, 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_backup] store location [/mounts/prod\_backup/my\_backup] is not accessible on the node [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; nested: FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_backup] store location [/mounts/prod\_backup/my\_backup] is not accessible on the node [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; nested: FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my\_backup] store location [/mounts/prod\_backup/my\_backup] is not accessible on the node [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; nested: FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw (Permission denied)]; ']]]; ", > > > "status" : 500 > > > } > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > The only thing I can think of is that file permissions are wrong. But I'm not sure how to fix that. > > > > On the two nodes not listed in the error, the elasticsearch gid and uid are different that on the other three nodes. So I can't just chown -R elasticsearch:elasticsearch. > > > > Do I have to make sure that the uid and gid are the same on all nodes? How am I supposed to do that when I install from the apt repositories? > > > > Also, log-estools-01 has data.node = false. Why does it still need the prod\_backup mount? > > > > log-estools-01 is on Ubuntu 14.04. > > > > Everything else is Ubuntu 12.04. > > > > Running ES 1.4.4. > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 17, 2015, 12:36am UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/8 "2015-03-17T00:36:14Z")
iSCSI is different, you shouldn't run into this. On 16 March 2015 at 17:20, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > I’m not that familiar with iSCSI so I hesitate to say for sure, but > anytime you are cross-mounting filesystems on Linux you have to take > uid/gid consistency into account. > > - Andrew > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > Would an iSCSI mount have the same issue? I believe our SAN supports both. > > --David Reagan > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > Hi David, > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > identical uid/gid mappings across all machines. It’s just one of those > > annoying sys-admin tasks that one has to take into account when using NFS. > > To get your permissions back to less permissive settings you will have to > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > for more context. > > > > - Andrew > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > First, it is a file permissions issue. I did get snapshots to run when I > > chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > "How can I get snapshots working without using insecure file permissions?" > > > > root@log-elasticsearch-01:~# mount > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > none on /sys/fs/fuse/connections type fusectl (rw) > > none on /sys/kernel/debug type debugfs (rw) > > none on /sys/kernel/security type securityfs (rw) > > udev on /dev type devtmpfs (rw,mode=0755) > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > /dev/sda1 on /boot type ext2 (rw) > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type > > nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > /mounts/prod\_backup/ > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > wrote: > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > /mounts/prod\_backup_? > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > Why does this happen? > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d > > > > > > > > > '{ > > > > > > > > > > > ``` > > > > > > "type": "fs", > > > > > > "settings": { > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > "compress": true > > > > > > } > > > > > > > > > > > > ``` > > > > > > > > > > > > }' > > > > > > { > > > > > > "error" : > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > nested: > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > nested: > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > nested: > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > (Permission denied)]; ']]]; ", > > > > > > "status" : 500 > > > > > > } > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > The only thing I can think of is that file permissions are wrong. But > > > > I'm not sure how to fix that. > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and uid > > > > are different that on the other three nodes. So I can't just _chown -R > > > > elasticsearch:elasticsearch_. > > > > > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > > > How am I supposed to do that when I install from the apt repositories? > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > > the prod\_backup mount? > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > Running ES 1.4.4. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9iSiygPu11vq%2BOXe2%2BhZbCtx\_P5vWQqp6QrHMabd6gNg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9iSiygPu11vq%2BOXe2%2BhZbCtx_P5vWQqp6QrHMabd6gNg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 17, 2015, 12:37am UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/9 "2015-03-17T00:37:07Z")
If I were manually creating the elasticsearch user, that'd be easy. But I'm relying on apt to do the job for me. So, yeah... Hmm... I suppose I could manually create an elasticsearch2 user, then modify the defaults files to use it when running ES. Still seems clunky... --David Reagan On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > I’m not that familiar with iSCSI so I hesitate to say for sure, but > anytime you are cross-mounting filesystems on Linux you have to take > uid/gid consistency into account. > > - Andrew > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > Would an iSCSI mount have the same issue? I believe our SAN supports both. > > --David Reagan > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > Hi David, > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > identical uid/gid mappings across all machines. It’s just one of those > > annoying sys-admin tasks that one has to take into account when using NFS. > > To get your permissions back to less permissive settings you will have to > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > for more context. > > > > - Andrew > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > First, it is a file permissions issue. I did get snapshots to run when I > > chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > "How can I get snapshots working without using insecure file permissions?" > > > > root@log-elasticsearch-01:~# mount > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > none on /sys/fs/fuse/connections type fusectl (rw) > > none on /sys/kernel/debug type debugfs (rw) > > none on /sys/kernel/security type securityfs (rw) > > udev on /dev type devtmpfs (rw,mode=0755) > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > /dev/sda1 on /boot type ext2 (rw) > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type > > nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > /mounts/prod\_backup/ > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > wrote: > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > /mounts/prod\_backup_? > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > Why does this happen? > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d > > > > > > > > > '{ > > > > > > > > > > > ``` > > > > > > "type": "fs", > > > > > > "settings": { > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > "compress": true > > > > > > } > > > > > > > > > > > > ``` > > > > > > > > > > > > }' > > > > > > { > > > > > > "error" : > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > nested: > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > nested: > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > nested: > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > (Permission denied)]; ']]]; ", > > > > > > "status" : 500 > > > > > > } > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > The only thing I can think of is that file permissions are wrong. But > > > > I'm not sure how to fix that. > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and uid > > > > are different that on the other three nodes. So I can't just _chown -R > > > > elasticsearch:elasticsearch_. > > > > > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > > > How am I supposed to do that when I install from the apt repositories? > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > > the prod\_backup mount? > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > Running ES 1.4.4. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdAhoF\_yWEL3e\_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdAhoF_yWEL3e_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 17, 2015, 4:00pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/10 "2015-03-17T16:00:59Z")
@Mark Walkom, So, I'm looking into iscsi. From what I have learned so far, you actually format the LUN with whatever file system you want. So, wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or ext4? Since Ubuntu would treat it like a normal partition and use typical linux file perms on it. --David Reagan On Mon, Mar 16, 2015 at 5:37 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > If I were manually creating the elasticsearch user, that'd be easy. But > I'm relying on apt to do the job for me. So, yeah... > > Hmm... I suppose I could manually create an elasticsearch2 user, then > modify the defaults files to use it when running ES. Still seems clunky... > > --David Reagan > > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > I’m not that familiar with iSCSI so I hesitate to say for sure, but > > anytime you are cross-mounting filesystems on Linux you have to take > > uid/gid consistency into account. > > > > - Andrew > > > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > Would an iSCSI mount have the same issue? I believe our SAN supports > > both. > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > > > Hi David, > > > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > > identical uid/gid mappings across all machines. It’s just one of those > > > annoying sys-admin tasks that one has to take into account when using NFS. > > > To get your permissions back to less permissive settings you will have to > > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > > for more context. > > > > > > - Andrew > > > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > First, it is a file permissions issue. I did get snapshots to run when I > > > chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > > "How can I get snapshots working without using insecure file permissions?" > > > > > > root@log-elasticsearch-01:~# mount > > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > none on /sys/fs/fuse/connections type fusectl (rw) > > > none on /sys/kernel/debug type debugfs (rw) > > > none on /sys/kernel/security type securityfs (rw) > > > udev on /dev type devtmpfs (rw,mode=0755) > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > > /dev/sda1 on /boot type ext2 (rw) > > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 > > > type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > > /mounts/prod\_backup/ > > > > > > --David Reagan > > > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > > wrote: > > > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > > /mounts/prod\_backup_? > > > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > Why does this happen? > > > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d > > > > > > > > > > > '{ > > > > > > > > > > > > > ``` > > > > > > > "type": "fs", > > > > > > > "settings": { > > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > > "compress": true > > > > > > > } > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > }' > > > > > > > { > > > > > > > "error" : > > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > > nested: > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > > nested: > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > > nested: > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > > (Permission denied)]; ']]]; ", > > > > > > > "status" : 500 > > > > > > > } > > > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > > > The only thing I can think of is that file permissions are wrong. But > > > > > I'm not sure how to fix that. > > > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and > > > > > uid are different that on the other three nodes. So I can't just _chown > > > > > -R elasticsearch:elasticsearch_. > > > > > > > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > > > > How am I supposed to do that when I install from the apt repositories? > > > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > > > the prod\_backup mount? > > > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > > > Running ES 1.4.4. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 17, 2015, 6:06pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/11 "2015-03-17T18:06:48Z")
As has been mentioned, use uid remapping when mounting. On 16 March 2015 at 17:37, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > If I were manually creating the elasticsearch user, that'd be easy. But > I'm relying on apt to do the job for me. So, yeah... > > Hmm... I suppose I could manually create an elasticsearch2 user, then > modify the defaults files to use it when running ES. Still seems clunky... > > --David Reagan > > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > I’m not that familiar with iSCSI so I hesitate to say for sure, but > > anytime you are cross-mounting filesystems on Linux you have to take > > uid/gid consistency into account. > > > > - Andrew > > > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > Would an iSCSI mount have the same issue? I believe our SAN supports > > both. > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > > > Hi David, > > > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > > identical uid/gid mappings across all machines. It’s just one of those > > > annoying sys-admin tasks that one has to take into account when using NFS. > > > To get your permissions back to less permissive settings you will have to > > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > > for more context. > > > > > > - Andrew > > > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > First, it is a file permissions issue. I did get snapshots to run when I > > > chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > > "How can I get snapshots working without using insecure file permissions?" > > > > > > root@log-elasticsearch-01:~# mount > > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > none on /sys/fs/fuse/connections type fusectl (rw) > > > none on /sys/kernel/debug type debugfs (rw) > > > none on /sys/kernel/security type securityfs (rw) > > > udev on /dev type devtmpfs (rw,mode=0755) > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > > /dev/sda1 on /boot type ext2 (rw) > > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 > > > type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > > /mounts/prod\_backup/ > > > > > > --David Reagan > > > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > > wrote: > > > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > > /mounts/prod\_backup_? > > > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > Why does this happen? > > > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' -d > > > > > > > > > > > '{ > > > > > > > > > > > > > ``` > > > > > > > "type": "fs", > > > > > > > "settings": { > > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > > "compress": true > > > > > > > } > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > }' > > > > > > > { > > > > > > > "error" : > > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > > nested: > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > > nested: > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > > nested: > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > > (Permission denied)]; ']]]; ", > > > > > > > "status" : 500 > > > > > > > } > > > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > > > The only thing I can think of is that file permissions are wrong. But > > > > > I'm not sure how to fix that. > > > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and > > > > > uid are different that on the other three nodes. So I can't just _chown > > > > > -R elasticsearch:elasticsearch_. > > > > > > > > > > Do I have to make sure that the uid and gid are the same on all nodes? > > > > > How am I supposed to do that when I install from the apt repositories? > > > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > > > the prod\_backup mount? > > > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > > > Running ES 1.4.4. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdAhoF\_yWEL3e\_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdAhoF_yWEL3e_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdAhoF\_yWEL3e\_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdAhoF_yWEL3e_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_t0%2Bt3O1bxRu5K7g%2BwjWNiqSmQvcHcYkWKPXQbS96aXw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_t0%2Bt3O1bxRu5K7g%2BwjWNiqSmQvcHcYkWKPXQbS96aXw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 17, 2015, 6:11pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/12 "2015-03-17T18:11:59Z")
iSCSI can be mounted as a block device that you can format however you want, if you do it that way the uid problem won't show up as the system sees it as a local FS. On 17 March 2015 at 09:00, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > @Mark Walkom, So, I'm looking into iscsi. From what I have learned so far, > you actually format the LUN with whatever file system you want. So, > wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or > ext4? Since Ubuntu would treat it like a normal partition and use typical > linux file perms on it. > > --David Reagan > > On Mon, Mar 16, 2015 at 5:37 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > If I were manually creating the elasticsearch user, that'd be easy. But > > I'm relying on apt to do the job for me. So, yeah... > > > > Hmm... I suppose I could manually create an elasticsearch2 user, then > > modify the defaults files to use it when running ES. Still seems clunky... > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > > > I’m not that familiar with iSCSI so I hesitate to say for sure, but > > > anytime you are cross-mounting filesystems on Linux you have to take > > > uid/gid consistency into account. > > > > > > - Andrew > > > > > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > Would an iSCSI mount have the same issue? I believe our SAN supports > > > both. > > > > > > --David Reagan > > > > > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > > > wrote: > > > > > > > Hi David, > > > > > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > > > identical uid/gid mappings across all machines. It’s just one of those > > > > annoying sys-admin tasks that one has to take into account when using NFS. > > > > To get your permissions back to less permissive settings you will have to > > > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > > > for more context. > > > > > > > > - Andrew > > > > > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > First, it is a file permissions issue. I did get snapshots to run when > > > > I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > > > "How can I get snapshots working without using insecure file permissions?" > > > > > > > > root@log-elasticsearch-01:~# mount > > > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > > none on /sys/fs/fuse/connections type fusectl (rw) > > > > none on /sys/kernel/debug type debugfs (rw) > > > > none on /sys/kernel/security type securityfs (rw) > > > > udev on /dev type devtmpfs (rw,mode=0755) > > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > > > /dev/sda1 on /boot type ext2 (rw) > > > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 > > > > type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > > > /mounts/prod\_backup/ > > > > > > > > --David Reagan > > > > > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > > > wrote: > > > > > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > > > /mounts/prod\_backup_? > > > > > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > > > Why does this happen? > > > > > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' > > > > > > > > > > > > > -d '{ > > > > > > > > > > > > > > > ``` > > > > > > > > "type": "fs", > > > > > > > > "settings": { > > > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > > > "compress": true > > > > > > > > } > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > }' > > > > > > > > { > > > > > > > > "error" : > > > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > > > nested: > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > > > nested: > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > > > nested: > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > > > (Permission denied)]; ']]]; ", > > > > > > > > "status" : 500 > > > > > > > > } > > > > > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > > > > > The only thing I can think of is that file permissions are wrong. But > > > > > > I'm not sure how to fix that. > > > > > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and > > > > > > uid are different that on the other three nodes. So I can't just _chown > > > > > > -R elasticsearch:elasticsearch_. > > > > > > > > > > > > Do I have to make sure that the uid and gid are the same on all > > > > > > nodes? How am I supposed to do that when I install from the apt > > > > > > repositories? > > > > > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > > > > the prod\_backup mount? > > > > > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > > > > > Running ES 1.4.4. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF\_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 17, 2015, 7:10pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/13 "2015-03-17T19:10:31Z")
According to [NFS uid/gid mapping – KernelCrash](http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/) the method described in that post only applies to old, out of date, systems. I also found no mention of a map file in [http://linux.die.net/man/8/mount.nfs](http://linux.die.net/man/8/mount.nfs) or [http://linux.die.net/man/5/nfs](http://linux.die.net/man/5/nfs) The closest I found to something I could use was > But it seems to only apply to nfs version 4. We, for some reason, are on version 3. Hmm... Would adding the suid flag to the mount help? As for iSCSI, it doesn't matter if the file system sees it as a local device. Currently my file system sees my nfs mounts as pretty much local mounts. But it still thinks that some of my elasticsearch owned files are actually owned by my ntp user... I don't see how "formatting however I want" will help with that kind of issue. Permissions are set by uid and gid, not the name. Unless iSCSI has some feature that overrides that. --David Reagan On Tue, Mar 17, 2015 at 11:11 AM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > iSCSI can be mounted as a block device that you can format however you > want, if you do it that way the uid problem won't show up as the system > sees it as a local FS. > > On 17 March 2015 at 09:00, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > @Mark Walkom, So, I'm looking into iscsi. From what I have learned so > > far, you actually format the LUN with whatever file system you want. So, > > wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or > > ext4? Since Ubuntu would treat it like a normal partition and use typical > > linux file perms on it. > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 5:37 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > If I were manually creating the elasticsearch user, that'd be easy. But > > > I'm relying on apt to do the job for me. So, yeah... > > > > > > Hmm... I suppose I could manually create an elasticsearch2 user, then > > > modify the defaults files to use it when running ES. Still seems clunky... > > > > > > --David Reagan > > > > > > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > > > wrote: > > > > > > > I’m not that familiar with iSCSI so I hesitate to say for sure, but > > > > anytime you are cross-mounting filesystems on Linux you have to take > > > > uid/gid consistency into account. > > > > > > > > - Andrew > > > > > > > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > Would an iSCSI mount have the same issue? I believe our SAN supports > > > > both. > > > > > > > > --David Reagan > > > > > > > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > > > > wrote: > > > > > > > > > Hi David, > > > > > > > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > > > > identical uid/gid mappings across all machines. It’s just one of those > > > > > annoying sys-admin tasks that one has to take into account when using NFS. > > > > > To get your permissions back to less permissive settings you will have to > > > > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > > > > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > > > > for more context. > > > > > > > > > > - Andrew > > > > > > > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > > First, it is a file permissions issue. I did get snapshots to run when > > > > > I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > > > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > > > > "How can I get snapshots working without using insecure file permissions?" > > > > > > > > > > root@log-elasticsearch-01:~# mount > > > > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > > > none on /sys/fs/fuse/connections type fusectl (rw) > > > > > none on /sys/kernel/debug type debugfs (rw) > > > > > none on /sys/kernel/security type securityfs (rw) > > > > > udev on /dev type devtmpfs (rw,mode=0755) > > > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > > > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > > > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > > > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > > > > /dev/sda1 on /boot type ext2 (rw) > > > > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > > > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > > > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 > > > > > type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > > > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > > > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > > > > /mounts/prod\_backup/ > > > > > > > > > > --David Reagan > > > > > > > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > > > > wrote: > > > > > > > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > > > > /mounts/prod\_backup_? > > > > > > > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > > > > > Why does this happen? > > > > > > > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' > > > > > > > > > > > > > > > -d '{ > > > > > > > > > > > > > > > > > ``` > > > > > > > > > "type": "fs", > > > > > > > > > "settings": { > > > > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > > > > "compress": true > > > > > > > > > } > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > }' > > > > > > > > > { > > > > > > > > > "error" : > > > > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > > > > nested: > > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > > > > nested: > > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > > > > nested: > > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > > > > (Permission denied)]; ']]]; ", > > > > > > > > > "status" : 500 > > > > > > > > > } > > > > > > > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > > > > > > > The only thing I can think of is that file permissions are wrong. > > > > > > > But I'm not sure how to fix that. > > > > > > > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and > > > > > > > uid are different that on the other three nodes. So I can't just _chown > > > > > > > -R elasticsearch:elasticsearch_. > > > > > > > > > > > > > > Do I have to make sure that the uid and gid are the same on all > > > > > > > nodes? How am I supposed to do that when I install from the apt > > > > > > > repositories? > > > > > > > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still > > > > > > > need the prod\_backup mount? > > > > > > > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > > > > > > > Running ES 1.4.4. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF\_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF\_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Aeunrn-ga5WkNcfq61G\_gEEAtxi-hocsFjf6cxmSAD2pg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Aeunrn-ga5WkNcfq61G_gEEAtxi-hocsFjf6cxmSAD2pg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 18, 2015, 7:28am UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/14 "2015-03-18T07:28:08Z")
On Tuesday, March 17, 2015 at 01:37 CET, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > wrote: > > > I'm not that familiar with iSCSI so I hesitate to say for sure, but > > anytime you are cross-mounting filesystems on Linux you have to take > > uid/gid consistency into account. > > If I were manually creating the elasticsearch user, that'd be easy. > But I'm relying on apt to do the job for me. So, yeah... You can create the elasticsearch user with the uid you want before the Debian package gets a chance to do it. > Hmm... I suppose I could manually create an elasticsearch2 user, > then modify the defaults files to use it when running ES. Still > seems clunky... No need. If the elasticsearch user exists the package installation will just skip that step. -- Magnus Bäck | Software Engineer, Development Tools [magnus.back@sonymobile.com](mailto:magnus.back@sonymobile.com) | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20150318072807.GC15152%40seldlx20533.corpusers.net](https://groups.google.com/d/msgid/elasticsearch/20150318072807.GC15152%40seldlx20533.corpusers.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [March 18, 2015, 12:21pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/15 "2015-03-18T12:21:19Z")
yes, that's the difference between a (network service which exposes ) block storage like iSCSI and a network file system like NFS ( or glusterFS or Lustre... ). I don't see why on a local device (iSCSI) you'd have any issue with numeric uid not matching the 'name' of the user - unless, of course, you were to detach the volume from host/vm-1 and attached it to another system which had a different user table. centralised authentication systems like ldap would solve this for you too 🙂 (though it seems overkill when all you'd have to do is plan your uid space properly across client + nfs servers ...). configuration systems like puppet / chef / ansible / salt should also help you ensure uids across multiple system... Anyway, Magnus' suggestion is on the money for the '1 server problem' On Wed, Mar 18, 2015 at 3:00 AM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > @Mark Walkom, So, I'm looking into iscsi. From what I have learned so far, > you actually format the LUN with whatever file system you want. So, > wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or > ext4? Since Ubuntu would treat it like a normal partition and use typical > linux file perms on it. > > --David Reagan > > On Mon, Mar 16, 2015 at 5:37 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > If I were manually creating the elasticsearch user, that'd be easy. But > > I'm relying on apt to do the job for me. So, yeah... > > > > Hmm... I suppose I could manually create an elasticsearch2 user, then > > modify the defaults files to use it when running ES. Still seems clunky... > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) wrote: > > > > > I’m not that familiar with iSCSI so I hesitate to say for sure, but > > > anytime you are cross-mounting filesystems on Linux you have to take > > > uid/gid consistency into account. > > > > > > - Andrew > > > > > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > Would an iSCSI mount have the same issue? I believe our SAN supports > > > both. > > > > > > --David Reagan > > > > > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > > > wrote: > > > > > > > Hi David, > > > > > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > > > identical uid/gid mappings across all machines. It’s just one of those > > > > annoying sys-admin tasks that one has to take into account when using NFS. > > > > To get your permissions back to less permissive settings you will have to > > > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > > > > > See [http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > > > for more context. > > > > > > > > - Andrew > > > > > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > First, it is a file permissions issue. I did get snapshots to run when > > > > I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > > > "How can I get snapshots working without using insecure file permissions?" > > > > > > > > root@log-elasticsearch-01:~# mount > > > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > > none on /sys/fs/fuse/connections type fusectl (rw) > > > > none on /sys/kernel/debug type debugfs (rw) > > > > none on /sys/kernel/security type securityfs (rw) > > > > udev on /dev type devtmpfs (rw,mode=0755) > > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > > > /dev/sda1 on /boot type ext2 (rw) > > > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 > > > > type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > > > /mounts/prod\_backup/ > > > > > > > > --David Reagan > > > > > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > > > wrote: > > > > > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > > > /mounts/prod\_backup_? > > > > > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > > > Why does this happen? > > > > > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' > > > > > > > > > > > > > -d '{ > > > > > > > > > > > > > > > ``` > > > > > > > > "type": "fs", > > > > > > > > "settings": { > > > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > > > "compress": true > > > > > > > > } > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > }' > > > > > > > > { > > > > > > > > "error" : > > > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > > > nested: > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > > > nested: > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > > > nested: > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > > > (Permission denied)]; ']]]; ", > > > > > > > > "status" : 500 > > > > > > > > } > > > > > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > > > > > The only thing I can think of is that file permissions are wrong. But > > > > > > I'm not sure how to fix that. > > > > > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and > > > > > > uid are different that on the other three nodes. So I can't just _chown > > > > > > -R elasticsearch:elasticsearch_. > > > > > > > > > > > > Do I have to make sure that the uid and gid are the same on all > > > > > > nodes? How am I supposed to do that when I install from the apt > > > > > > repositories? > > > > > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still need > > > > > > the prod\_backup mount? > > > > > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > > > > > Running ES 1.4.4. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 18, 2015, 4:55pm UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/16 "2015-03-18T16:55:47Z")
> unless, of course, you were to detach the volume from host/vm-1 and > attached it to another system which had a different user table. > This is exactly the problem. For snapshot repositories, your repository > directory has to be accessible by all your data nodes. Thus, I mount the > same folder, in the same location, on all my data nodes. Thus, my uid/gid > issues. Which would also apply on iSCSI. Anyway, I modified my Ansible elasticsearch role to support creating a custom user and group. Tested it on my tiny test cluster, and it seems to work ok. Now I just need to run it on my prod cluster and deal with a full cluster restart. _sigh_ --David Reagan On Wed, Mar 18, 2015 at 5:21 AM, Norberto Meijome [numard@gmail.com](mailto:numard@gmail.com) wrote: > yes, that's the difference between a (network service which exposes ) > block storage like iSCSI and a network file system like NFS ( or glusterFS > or Lustre... ). > > I don't see why on a local device (iSCSI) you'd have any issue with > numeric uid not matching the 'name' of the user - unless, of course, you > were to detach the volume from host/vm-1 and attached it to another system > which had a different user table. > > centralised authentication systems like ldap would solve this for you too > 🙂 (though it seems overkill when all you'd have to do is plan your uid > space properly across client + nfs servers ...). configuration systems like > puppet / chef / ansible / salt should also help you ensure uids across > multiple system... > > Anyway, Magnus' suggestion is on the money for the '1 server problem' > > On Wed, Mar 18, 2015 at 3:00 AM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > @Mark Walkom, So, I'm looking into iscsi. From what I have learned so > > far, you actually format the LUN with whatever file system you want. So, > > wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or > > ext4? Since Ubuntu would treat it like a normal partition and use typical > > linux file perms on it. > > > > --David Reagan > > > > On Mon, Mar 16, 2015 at 5:37 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > If I were manually creating the elasticsearch user, that'd be easy. But > > > I'm relying on apt to do the job for me. So, yeah... > > > > > > Hmm... I suppose I could manually create an elasticsearch2 user, then > > > modify the defaults files to use it when running ES. Still seems clunky... > > > > > > --David Reagan > > > > > > On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > > > wrote: > > > > > > > I’m not that familiar with iSCSI so I hesitate to say for sure, but > > > > anytime you are cross-mounting filesystems on Linux you have to take > > > > uid/gid consistency into account. > > > > > > > > - Andrew > > > > > > > > On Mar 16, 2015, at 4:46 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > Would an iSCSI mount have the same issue? I believe our SAN supports > > > > both. > > > > > > > > --David Reagan > > > > > > > > On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden [andrew@elastic.co](mailto:andrew@elastic.co) > > > > wrote: > > > > > > > > > Hi David, > > > > > > > > > > This is a common problem with NFS. Unfortunately the protocol assumes > > > > > identical uid/gid mappings across all machines. It’s just one of those > > > > > annoying sys-admin tasks that one has to take into account when using NFS. > > > > > To get your permissions back to less permissive settings you will have to > > > > > edit the /etc/passwd and /etc/group files to keep them in sync. > > > > > > > > > > See [Troubleshooting](http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4) > > > > > for more context. > > > > > > > > > > - Andrew > > > > > > > > > > On Mar 16, 2015, at 4:04 PM, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > > First, it is a file permissions issue. I did get snapshots to run when > > > > > I chmoded to 777. As you can see from the ls output, /mounts/prod\_backup is > > > > > 777. Prior to that it was 775 or 755. So, I could revise my question to > > > > > "How can I get snapshots working without using insecure file permissions?" > > > > > > > > > > root@log-elasticsearch-01:~# mount > > > > > /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) > > > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > > > none on /sys/fs/fuse/connections type fusectl (rw) > > > > > none on /sys/kernel/debug type debugfs (rw) > > > > > none on /sys/kernel/security type securityfs (rw) > > > > > udev on /dev type devtmpfs (rw,mode=0755) > > > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) > > > > > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) > > > > > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) > > > > > none on /run/shm type tmpfs (rw,nosuid,nodev) > > > > > /dev/sda1 on /boot type ext2 (rw) > > > > > rpc\_pipefs on /run/rpc\_pipefs type rpc\_pipefs (rw) > > > > > nfsip:/vol/Logs/prod\_backup on /mounts/prod\_backup type nfs > > > > > (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 > > > > > type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) > > > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts > > > > > drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts > > > > > > > > > > root@log-elasticsearch-01:~# ls -ld /mounts/prod\_backup/ > > > > > drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 > > > > > /mounts/prod\_backup/ > > > > > > > > > > --David Reagan > > > > > > > > > > On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > > > > wrote: > > > > > > > > > > > Can you post the output from _mount_ and _ls -ld /mounts > > > > > > /mounts/prod\_backup_? > > > > > > > > > > > > On 16 March 2015 at 13:33, David Reagan [jerrac@gmail.com](mailto:jerrac@gmail.com) wrote: > > > > > > > > > > > > > Why does this happen? > > > > > > > > > > > > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup?pretty=true](http://localhost:9200/_snapshot/my_backup?pretty=true)' > > > > > > > > > > > > > > > -d '{ > > > > > > > > > > > > > > > > > ``` > > > > > > > > > "type": "fs", > > > > > > > > > "settings": { > > > > > > > > > "location": "/mounts/prod_backup/my_backup", > > > > > > > > > "compress": true > > > > > > > > > } > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > }' > > > > > > > > > { > > > > > > > > > "error" : > > > > > > > > > "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] > > > > > > > > > [vxUQwUTCQwOaLyCy0eMK8A, > > > > > > > > > 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > > [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; > > > > > > > > > nested: > > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-vxUQwUTCQwOaLyCy0eMK8A > > > > > > > > > (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, > > > > > > > > > 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > > [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; > > > > > > > > > nested: > > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ > > > > > > > > > (Permission denied)]; '], [ffpuQF\_zRZGGPRkZRgq1mw, > > > > > > > > > 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; > > > > > > > > > nested: RepositoryVerificationException[[my\_backup] store location > > > > > > > > > [/mounts/prod\_backup/my\_backup] is not accessible on the node > > > > > > > > > [[log-elasticsearch-03][ffpuQF\_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; > > > > > > > > > nested: > > > > > > > > > FileNotFoundException[/mounts/prod\_backup/my\_backup/tests-yZ57gviiQUGS55tr\_ULhhg-ffpuQF\_zRZGGPRkZRgq1mw > > > > > > > > > (Permission denied)]; ']]]; ", > > > > > > > > > "status" : 500 > > > > > > > > > } > > > > > > > > > > > > > > /mounts/prod\_backup is an NFS share mounted on all my ES nodes. > > > > > > > > > > > > > > The only thing I can think of is that file permissions are wrong. > > > > > > > But I'm not sure how to fix that. > > > > > > > > > > > > > > On the two nodes not listed in the error, the elasticsearch gid and > > > > > > > uid are different that on the other three nodes. So I can't just _chown > > > > > > > -R elasticsearch:elasticsearch_. > > > > > > > > > > > > > > Do I have to make sure that the uid and gid are the same on all > > > > > > > nodes? How am I supposed to do that when I install from the apt > > > > > > > repositories? > > > > > > > > > > > > > > Also, log-estools-01 has _data.node = false_. Why does it still > > > > > > > need the prod\_backup mount? > > > > > > > > > > > > > > log-estools-01 is on Ubuntu 14.04. > > > > > > > > > > > > > > Everything else is Ubuntu 12.04. > > > > > > > > > > > > > > Running ES 1.4.4. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H\_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q\_w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7\_xjV4eAcg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co) > > > > > [https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co) > > > > [https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Norberto 'Beto' Meijome > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/\_1ClJTJOEcU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANo%2B\_AcPMQ2W5tffd3cMCfuvjH%2BAfuk9U9GiPeLMAGvN2G\_gGQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AcPMQ2W5tffd3cMCfuvjH%2BAfuk9U9GiPeLMAGvN2G_gGQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:25am UTC](https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/17 "2017-07-06T00:25:48Z")