Hi,
We're getting an intermittent AccessDeniedException when creating snapshots on our cluster. As far as I can see all permissions are correct and config is the same across all servers.
ES Version: 7.6.0
The /home/shopfiles
directory is a shared NFS mount across all servers.
The exception:
RepositoryException[[shopfiles] cannot create blob store]; nested: AccessControlException[access denied ("java.io.FilePermission" "/home/shopfiles/private/elasticsearch" "read")];
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:424)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.shardContainer(BlobStoreRepository.java:987)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.shardContainer(BlobStoreRepository.java:983)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:1468)
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:340)
at org.elasticsearch.snapshots.SnapshotShardsService.lambda$startNewShards$1(SnapshotShardsService.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/shopfiles/private/elasticsearch" "read")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
at java.base/sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:295)
at java.base/java.nio.file.Files.createDirectories(Files.java:765)
at org.elasticsearch.common.blobstore.fs.FsBlobStore.<init>(FsBlobStore.java:46)
at org.elasticsearch.repositories.fs.FsRepository.createBlobStore(FsRepository.java:119)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:420)
... 9 more
Config on all servers:
root@es1 /e/elasticsearch# cat /etc/elasticsearch/elasticsearch.yml | grep path.repo
path.repo: ["/home/shopfiles/private/elasticsearch/snapshots"]
Listing directory mentioned in exception as Elasticsearch user:
root@es1 /e/elasticsearch# sudo -u elasticsearch -H ls -al /home/shopfiles/private/elasticsearch
total 32
drwxrwxr-x 8 myuser myuser 4096 Dec 19 2018 .
drwxr-xr-x 5 myuser myuser 4096 Jan 31 21:49 ..
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar 3 10:15 snapshots
I have checked the UID / GID of Elasticsearch on all nodes, they are the same.
root@es5 /e/elasticsearch# cat /etc/passwd | grep elasticsearch
elasticsearch:x:113:116::/nonexistent:/bin/false
Any ideas?