Following is our ES setup:
version : 2.4.1
Nodes: 2
Shared directory : /data/backups on master (also a data) node and mounted using sshfs (fuse) on other nodes.
About, sshfs mount,
- Below is the fstab entry on non-master data node:
<sshfs#user@node1-ip:/data/backups /data/backups fuse rw,gid=125,uid=117,umask=000,default_permissions,IdentityFile="/home/user/.ssh/id_rsa",allow_other 0 0/> - Note: gid:125 and uid:117 correspond to group 'elasticsearch' and user 'elasticsearch' respectively on the non-master data node.
On the mounted directory on non-master data nodes, I am able to create files/directories under /data/backups as user 'elasticsearch'.
But, still when we start a snapshot, we get the below error on the non-master data node.
<
[2018-06-27 15:13:30,495][WARN ][snapshots ] [Brainchild] [[company-1-1-network-element-health-2018.06.26][0]] [company-repo:radware-lb-26jun] failed to create snapshot
[company-1-1-network-element-health-2018.06.26][[company-1-1-network-element-health-2018.06.26][0]] IndexShardSnapshotFailedException[Failed to snapshot]; nested: ElasticsearchException[failed to create blob container]; nested: AccessDeniedException[/data/backups/indices/company-1-1-network-element-health-2018.06.26/0];
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:355)
at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:76)
at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:296)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchException[failed to create blob container]; nested: AccessDeniedException[/data/backups/indices/company-1-1-network-element-health-2018.06.26/0];
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:70)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.(BlobStoreIndexShardRepository.java:310)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.(BlobStoreIndexShardRepository.java:303)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.(BlobStoreIndexShardRepository.java:513)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:187)
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:340)
... 6 more
Caused by: java.nio.file.AccessDeniedException: /data/backups/indices/company-1-1-network-element-health-2018.06.26/0
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:86)
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:68)
... 11 more
/>
Output of ls command for the directory company-1-1-network-element-health-2018.06.26 within /data/backups.
<
indices/company-1-1-network-element-health-2018.06.26/:
total 12
drwxrwxrwx 1 elasticsearch elasticsearch 4096 Jun 27 15:14 1
-rwxrwxrwx 1 elasticsearch elasticsearch 4160 Jun 27 15:14 meta-radware-lb-26jun.dat
indices/company-1-1-network-element-health-2018.06.26/1:
total 153388
-rwxrwxrwx 1 elasticsearch elasticsearch 10815295 Jun 27 15:14 __0
-rwxrwxrwx 1 elasticsearch elasticsearch 2607854 Jun 27 15:14 __1
-rwxrwxrwx 1 elasticsearch elasticsearch 375 Jun 27 15:14 __10
-rwxrwxrwx 1 elasticsearch elasticsearch 8069937 Jun 27 15:14 __11
-rwxrwxrwx 1 elasticsearch elasticsearch 413 Jun 27 15:14 __12
: : :
-rwxrwxrwx 1 elasticsearch elasticsearch 937550 Jun 27 15:14 __s
-rwxrwxrwx 1 elasticsearch elasticsearch 2813 Jun 27 15:14 snap-radware-lb-26jun.dat
-rwxrwxrwx 1 elasticsearch elasticsearch 49489 Jun 27 15:14 __t
-rwxrwxrwx 1 elasticsearch elasticsearch 49489 Jun 27 15:14 __u
-rwxrwxrwx 1 elasticsearch elasticsearch 26853 Jun 27 15:14 __v
-rwxrwxrwx 1 elasticsearch elasticsearch 321 Jun 27 15:14 __w
-rwxrwxrwx 1 elasticsearch elasticsearch 3219485 Jun 27 15:14 __x
-rwxrwxrwx 1 elasticsearch elasticsearch 1251 Jun 27 15:14 __y
-rwxrwxrwx 1 elasticsearch elasticsearch 8421693 Jun 27 15:14 __z
/>
What am I missing here?
-Arun