This is a common problem with NFS. Unfortunately the protocol assumes identical uid/gid mappings across all machines. It’s just one of those annoying sys-admin tasks that one has to take into account when using NFS. To get your permissions back to less permissive settings you will have to edit the /etc/passwd and /etc/group files to keep them in sync.
On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:
First, it is a file permissions issue. I did get snapshots to run when I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is 777. Prior to that it was 775 or 755. So, I could revise my question to "How can I get snapshots working without using insecure file permissions?"
root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts
root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 /mounts/prod_backup/
--David Reagan
On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom <markwalkom@gmail.com mailto:markwalkom@gmail.com> wrote:
Can you post the output from mount and ls -ld /mounts /mounts/prod_backup?
On 16 March 2015 at 13:33, David Reagan <jerrac@gmail.com mailto:jerrac@gmail.com> wrote:
Why does this happen?
curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{
"type": "fs",
"settings": {
"location": "/mounts/prod_backup/my_backup",
"compress": true
}
}'
{
"error" : "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[my_backup] [vxUQwUTCQwOaLyCy0eMK8A, 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw (Permission denied)]; ']]]; ",
"status" : 500
}
/mounts/prod_backup is an NFS share mounted on all my ES nodes.
The only thing I can think of is that file permissions are wrong. But I'm not sure how to fix that.
On the two nodes not listed in the error, the elasticsearch gid and uid are different that on the other three nodes. So I can't just chown -R elasticsearch:elasticsearch.
Do I have to make sure that the uid and gid are the same on all nodes? How am I supposed to do that when I install from the apt repositories?
Also, log-estools-01 has data.node = false. Why does it still need the prod_backup mount?
log-estools-01 is on Ubuntu 14.04.
Everything else is Ubuntu 12.04.
Running ES 1.4.4.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.
(though it seems overkill when all you'd have to do is plan your uid