Hello,
I am hoping someone can help me with this problem I am having. We are on 1.7.1. The cluster consists of 12 hosts. Some of these hosts have 4 instances running, under users es1, es2, es3, es4. Some of the hosts have only one instance running, as root. The network drive is mounted on every, and verify command lists them all.
The repository is /backup1.
I created directory /backup1/syd, with 777 permissions on /backup1 and /backup1/syd and nobody:nobody owner. When I issue snapshot command, there's indices dir created, and under it, one dir for each index. The permissions on those are 755, root:root. The indices that have shards on hosts where elasticsearch is run as es1-es4 user failed to be backed up, since esX user can't create a directory under/backup1/syd/indices/INDEX_NAME (since root is the owner and esX doesn't have permissions to create dirs under it).
My question: what determines which user creates these directories? Why are they always created as root?
pwd
/backup1/syd
root@elasearch001.pod01:/backup1/syd
ll
total 156
-rw-r--r-- 1 root root 31 Jun 1 07:16 index
drwxr-xr-x 283 root root 28672 Jun 1 07:16 indices
-rw-r--r-- 1 root root 1822 Jun 1 07:16 metadata-snapshot_1
-rw-r--r-- 1 root root 118022 Jun 1 07:16 snapshot-snapshot_1
If I list info about my snapshot, I see:
"reason" : "IndexShardSnapshotFailedException[[sessions_20151124005110_2016_06_01][14] failed to list blobs]; nested: NoSuchFileException[/backup1/syd/indices/sessions_20151124005110_2016_06_01/14]; ",
This is because /backup1/syd/indices/sessions_20151124005110_2016_06_01/ is directory owned by root, with 755 permissions. User esX can't create directory 14 under it (and shard 14 happens to be on the host where elasticsearch is run as esX user).
Not sure how to solve this. I wanted to somehow set it to run as nonroot user, whose umask I could change, so it creates these directories with 777, so esX user can create sundirectories for each shard under it.
Thanks
Marjana