Why does creating a repository fail?

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But I'm
not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid are
different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes? How
am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need the
prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But I'm
not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid
are different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes? How
am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need the
prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But I'm
not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid
are different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need the
prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes identical uid/gid mappings across all machines. It’s just one of those annoying sys-admin tasks that one has to take into account when using NFS. To get your permissions back to less permissive settings you will have to edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is 777. Prior to that it was 775 or 755. So, I could revise my question to "How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 /mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom <markwalkom@gmail.com mailto:markwalkom@gmail.com> wrote:
Can you post the output from mount and ls -ld /mounts /mounts/prod_backup?

On 16 March 2015 at 13:33, David Reagan <jerrac@gmail.com mailto:jerrac@gmail.com> wrote:
Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" : "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[my_backup] [vxUQwUTCQwOaLyCy0eMK8A, 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw (Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid are different that on the other three nodes. So I can't just chown -R elasticsearch:elasticsearch.

Do I have to make sure that the uid and gid are the same on all nodes? How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co.
For more options, visit https://groups.google.com/d/optout.

You can do a uid remap for NFS mounts, eg
http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/

On 16 March 2015 at 16:04, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid
are different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_XqNm2S91VEMKCHpoxWzvxxLzHPBhC%3D66nNbSLhY%2BaoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Would an iSCSI mount have the same issue? I believe our SAN supports both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 for
more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid
are different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I’m not that familiar with iSCSI so I hesitate to say for sure, but anytime you are cross-mounting filesystems on Linux you have to take uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden <andrew@elastic.co mailto:andrew@elastic.co> wrote:
Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes identical uid/gid mappings across all machines. It’s just one of those annoying sys-admin tasks that one has to take into account when using NFS. To get your permissions back to less permissive settings you will have to edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan <jerrac@gmail.com mailto:jerrac@gmail.com> wrote:

First, it is a file permissions issue. I did get snapshots to run when I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is 777. Prior to that it was 775 or 755. So, I could revise my question to "How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 /mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom <markwalkom@gmail.com mailto:markwalkom@gmail.com> wrote:
Can you post the output from mount and ls -ld /mounts /mounts/prod_backup?

On 16 March 2015 at 13:33, David Reagan <jerrac@gmail.com mailto:jerrac@gmail.com> wrote:
Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" : "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[my_backup] [vxUQwUTCQwOaLyCy0eMK8A, 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] store location [/mounts/prod_backup/my_backup] is not accessible on the node [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; nested: FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw (Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid are different that on the other three nodes. So I can't just chown -R elasticsearch:elasticsearch.

Do I have to make sure that the uid and gid are the same on all nodes? How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co.
For more options, visit https://groups.google.com/d/optout.

iSCSI is different, you shouldn't run into this.

On 16 March 2015 at 17:20, Andrew Selden andrew@elastic.co wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d

'{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid
are different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9iSiygPu11vq%2BOXe2%2BhZbCtx_P5vWQqp6QrHMabd6gNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

If I were manually creating the elasticsearch user, that'd be easy. But I'm
relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d

'{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid
are different that on the other three nodes. So I can't just chown -R
elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdAhoF_yWEL3e_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

@Mark Walkom, So, I'm looking into iscsi. From what I have learned so far,
you actually format the LUN with whatever file system you want. So,
wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
ext4? Since Ubuntu would treat it like a normal partition and use typical
linux file perms on it.

--David Reagan

On Mon, Mar 16, 2015 at 5:37 PM, David Reagan jerrac@gmail.com wrote:

If I were manually creating the elasticsearch user, that'd be easy. But
I'm relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports
both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d

'{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and
uid are different that on the other three nodes. So I can't just chown
-R elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

As has been mentioned, use uid remapping when mounting.

On 16 March 2015 at 17:37, David Reagan jerrac@gmail.com wrote:

If I were manually creating the elasticsearch user, that'd be easy. But
I'm relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports
both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d

'{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and
uid are different that on the other three nodes. So I can't just chown
-R elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all nodes?
How am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdAhoF_yWEL3e_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdAhoF_yWEL3e_X0NSXJOR2ZwjS721%3DSwLA3-hQJtZ%2Bwg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_t0%2Bt3O1bxRu5K7g%2BwjWNiqSmQvcHcYkWKPXQbS96aXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

iSCSI can be mounted as a block device that you can format however you
want, if you do it that way the uid problem won't show up as the system
sees it as a local FS.

On 17 March 2015 at 09:00, David Reagan jerrac@gmail.com wrote:

@Mark Walkom, So, I'm looking into iscsi. From what I have learned so far,
you actually format the LUN with whatever file system you want. So,
wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
ext4? Since Ubuntu would treat it like a normal partition and use typical
linux file perms on it.

--David Reagan

On Mon, Mar 16, 2015 at 5:37 PM, David Reagan jerrac@gmail.com wrote:

If I were manually creating the elasticsearch user, that'd be easy. But
I'm relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports
both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co
wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when
I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true'

-d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and
uid are different that on the other three nodes. So I can't just chown
-R elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all
nodes? How am I supposed to do that when I install from the apt
repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

According to http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/
the method described in that post only applies to old, out of date,
systems.

I also found no mention of a map file in
http://linux.die.net/man/8/mount.nfs or http://linux.die.net/man/5/nfs

The closest I found to something I could use was


But it seems to only apply to nfs version 4. We, for some reason, are on
version 3.

Hmm... Would adding the suid flag to the mount help?

As for iSCSI, it doesn't matter if the file system sees it as a local
device. Currently my file system sees my nfs mounts as pretty much local
mounts. But it still thinks that some of my elasticsearch owned files are
actually owned by my ntp user... I don't see how "formatting however I
want" will help with that kind of issue. Permissions are set by uid and
gid, not the name. Unless iSCSI has some feature that overrides that.

--David Reagan

On Tue, Mar 17, 2015 at 11:11 AM, Mark Walkom markwalkom@gmail.com wrote:

iSCSI can be mounted as a block device that you can format however you
want, if you do it that way the uid problem won't show up as the system
sees it as a local FS.

On 17 March 2015 at 09:00, David Reagan jerrac@gmail.com wrote:

@Mark Walkom, So, I'm looking into iscsi. From what I have learned so
far, you actually format the LUN with whatever file system you want. So,
wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
ext4? Since Ubuntu would treat it like a normal partition and use typical
linux file perms on it.

--David Reagan

On Mon, Mar 16, 2015 at 5:37 PM, David Reagan jerrac@gmail.com wrote:

If I were manually creating the elasticsearch user, that'd be easy. But
I'm relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co
wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports
both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co
wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when
I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true'

-d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong.
But I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and
uid are different that on the other three nodes. So I can't just chown
-R elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all
nodes? How am I supposed to do that when I install from the apt
repositories?

Also, log-estools-01 has data.node = false. Why does it still
need the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2VPd9W5wLtSNeF_jwymt-Ac4BeVhdL6bZbHjRD1cFew%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Aeunrn-ga5WkNcfq61G_gEEAtxi-hocsFjf6cxmSAD2pg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

On Tuesday, March 17, 2015 at 01:37 CET,
David Reagan jerrac@gmail.com wrote:

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co
wrote:

I'm not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

If I were manually creating the elasticsearch user, that'd be easy.
But I'm relying on apt to do the job for me. So, yeah...

You can create the elasticsearch user with the uid you want before the
Debian package gets a chance to do it.

Hmm... I suppose I could manually create an elasticsearch2 user,
then modify the defaults files to use it when running ES. Still
seems clunky...

No need. If the elasticsearch user exists the package installation will
just skip that step.

--
Magnus Bäck | Software Engineer, Development Tools
magnus.back@sonymobile.com | Sony Mobile Communications

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150318072807.GC15152%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

yes, that's the difference between a (network service which exposes ) block
storage like iSCSI and a network file system like NFS ( or glusterFS or
Lustre... ).

I don't see why on a local device (iSCSI) you'd have any issue with numeric
uid not matching the 'name' of the user - unless, of course, you were to
detach the volume from host/vm-1 and attached it to another system which
had a different user table.

centralised authentication systems like ldap would solve this for you too
:slight_smile: (though it seems overkill when all you'd have to do is plan your uid
space properly across client + nfs servers ...). configuration systems like
puppet / chef / ansible / salt should also help you ensure uids across
multiple system...

Anyway, Magnus' suggestion is on the money for the '1 server problem'

On Wed, Mar 18, 2015 at 3:00 AM, David Reagan jerrac@gmail.com wrote:

@Mark Walkom, So, I'm looking into iscsi. From what I have learned so far,
you actually format the LUN with whatever file system you want. So,
wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
ext4? Since Ubuntu would treat it like a normal partition and use typical
linux file perms on it.

--David Reagan

On Mon, Mar 16, 2015 at 5:37 PM, David Reagan jerrac@gmail.com wrote:

If I were manually creating the elasticsearch user, that'd be easy. But
I'm relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports
both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co
wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when
I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true'

-d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But
I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and
uid are different that on the other three nodes. So I can't just chown
-R elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all
nodes? How am I supposed to do that when I install from the apt
repositories?

Also, log-estools-01 has data.node = false. Why does it still need
the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

unless, of course, you were to detach the volume from host/vm-1 and
attached it to another system which had a different user table.
This is exactly the problem. For snapshot repositories, your repository
directory has to be accessible by all your data nodes. Thus, I mount the
same folder, in the same location, on all my data nodes. Thus, my uid/gid
issues. Which would also apply on iSCSI.

Anyway, I modified my Ansible elasticsearch role to support creating a
custom user and group. Tested it on my tiny test cluster, and it seems to
work ok. Now I just need to run it on my prod cluster and deal with a full
cluster restart. sigh

--David Reagan

On Wed, Mar 18, 2015 at 5:21 AM, Norberto Meijome numard@gmail.com wrote:

yes, that's the difference between a (network service which exposes )
block storage like iSCSI and a network file system like NFS ( or glusterFS
or Lustre... ).

I don't see why on a local device (iSCSI) you'd have any issue with
numeric uid not matching the 'name' of the user - unless, of course, you
were to detach the volume from host/vm-1 and attached it to another system
which had a different user table.

centralised authentication systems like ldap would solve this for you too
:slight_smile: (though it seems overkill when all you'd have to do is plan your uid
space properly across client + nfs servers ...). configuration systems like
puppet / chef / ansible / salt should also help you ensure uids across
multiple system...

Anyway, Magnus' suggestion is on the money for the '1 server problem'

On Wed, Mar 18, 2015 at 3:00 AM, David Reagan jerrac@gmail.com wrote:

@Mark Walkom, So, I'm looking into iscsi. From what I have learned so
far, you actually format the LUN with whatever file system you want. So,
wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
ext4? Since Ubuntu would treat it like a normal partition and use typical
linux file perms on it.

--David Reagan

On Mon, Mar 16, 2015 at 5:37 PM, David Reagan jerrac@gmail.com wrote:

If I were manually creating the elasticsearch user, that'd be easy. But
I'm relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden andrew@elastic.co
wrote:

I’m not that familiar with iSCSI so I hesitate to say for sure, but
anytime you are cross-mounting filesystems on Linux you have to take
uid/gid consistency into account.

  • Andrew

On Mar 16, 2015, at 4:46 PM, David Reagan jerrac@gmail.com wrote:

Would an iSCSI mount have the same issue? I believe our SAN supports
both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden andrew@elastic.co
wrote:

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes
identical uid/gid mappings across all machines. It’s just one of those
annoying sys-admin tasks that one has to take into account when using NFS.
To get your permissions back to less permissive settings you will have to
edit the /etc/passwd and /etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
for more context.

  • Andrew

On Mar 16, 2015, at 4:04 PM, David Reagan jerrac@gmail.com wrote:

First, it is a file permissions issue. I did get snapshots to run when
I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom markwalkom@gmail.com
wrote:

Can you post the output from mount and ls -ld /mounts
/mounts/prod_backup
?

On 16 March 2015 at 13:33, David Reagan jerrac@gmail.com wrote:

Why does this happen?

curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true'

-d '{

"type": "fs",
"settings": {
    "location": "/mounts/prod_backup/my_backup",
    "compress": true
}

}'
{
"error" :
"RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
nested: RepositoryVerificationException[[my_backup]
[vxUQwUTCQwOaLyCy0eMK8A,
'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
(Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
(Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[my_backup] store location
[/mounts/prod_backup/my_backup] is not accessible on the node
[[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]];
nested:
FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
(Permission denied)]; ']]]; ",
"status" : 500
}

/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong.
But I'm not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and
uid are different that on the other three nodes. So I can't just chown
-R elasticsearch:elasticsearch
.

Do I have to make sure that the uid and gid are the same on all
nodes? How am I supposed to do that when I install from the apt
repositories?

Also, log-estools-01 has data.node = false. Why does it still
need the prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co
https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AcPMQ2W5tffd3cMCfuvjH%2BAfuk9U9GiPeLMAGvN2G_gGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.