Hi All
Having issue while taking cluster snapshot .
Env :
suse 12
5 node cluster 3M + 2D
elastic version : 5.5
Backup type : fs
Error :
"reason": "IndexShardSnapshotFailedException[Failed to snapshot]; nested: ElasticsearchException[failed to create blob container]; nested: AccessDeniedException[/backup/elasticsearch-backup/lb/indices/huUf9iZ9StCglJsOOsYs1A/0]; ",
"node_id": "BYIubOVHR9G23ehNfEfZ7Q",
"status": "INTERNAL_SERVER_ERROR"
Can somebody highlight or provide solution , why Access Denied exception is comming
Thanks
Please format your code using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
Please edit your post.
You probably have a stacktrace in logs. Could you share it ?
stacktrace
[2018-01-25T02:38:52,066][WARN ][o.e.s.SnapshotShardsService] [pc4cmelk02] [[.watcher-history-3-2017.12.26][0]] [lb_backup:lab_base1/pBNMVlaUSwGohP3nBsi0_g] failed to create snapshot
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: Failed to snapshot
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:397) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:88) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:335) [elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: org.elasticsearch.ElasticsearchException: failed to create blob container
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:70) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1078) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1071) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$SnapshotContext.<init>(BlobStoreRepository.java:1284) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:968) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:382) ~[elasticsearch-5.5.0.jar:5.5.0]
... 7 more
Caused by: java.nio.file.AccessDeniedException: /backup/elasticsearch-backup/lb/indices/u-zKaqTYSqWeJNe1vmBQDQ/0
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_101]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_101]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_101]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:86) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:68) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1078) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1071) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$SnapshotContext.<init>(BlobStoreRepository.java:1284) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:968) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:382) ~[elasticsearch-5.5.0.jar:5.5.0]
java.nio.file.AccessDeniedException: /backup/elasticsearch-backup/lb/indices/u-zKaqTYSqWeJNe1vmBQDQ/0
One of your nodes doesn't have access to this dir apparently?
When this error is appearing? What are the commands that you sent?
Hi
All the node have elasticsearch user permission to the repo path "/backup/elasticsearch-backup/lb "
while taking snaphot from "Cerebro v0.6.5" it creates the meta files ,but get the Access denied error while writing to the "index " folder
Thanks
Do you intentionally use this strange formatting for your text?
Could you ls -l your backup dir?
How many nodes you have?
Hi
ls -l output
ls -l /backup/elasticsearch-backup/lb/
-rw-r--r-- 1 488 488 29 Jan 23 09:20 incompatible-snapshots
-rw-r--r-- 1 488 488 29 Jan 25 02:37 index-1
-rw-r--r-- 1 488 488 5589 Jan 25 02:38 index-2
-rw-r--r-- 1 488 488 8 Jan 25 02:38 index.latest
drwxr-xr-x 56 488 488 8192 Jan 25 02:38 indices
-rw-r--r-- 1 488 488 15548 Jan 25 02:38 meta-pBNMVlaUSwGohP3nBsi0_g.dat
-rw-r--r-- 1 488 488 19886 Jan 25 02:38 snap-pBNMVlaUSwGohP3nBsi0_g.dat
Thanks
How many nodes you have?
Hi
5 nodes = 2 Data Node and 3 Master Node and X-Pack on top it .
Thanks
Himanshu
Is /backup/elasticsearch-backup/lb/ mount on every node and accessible via the user who is running the elasticsearch node locally.
Here I guess that one of the 5 nodes can not write there.
Hi
All the node have same mount point and elasticsearch user have permision on it.
Thanks
Is elasticsearch user actually user 488 as I can see in the ls -l?
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.