Snapshot create [Index Shard Snapshot Failed ] :: Acess Denied


#1
Hi All 

Having issue while taking cluster snapshot  . 
Env :  
suse 12 
5 node cluster  3M + 2D
elastic version : 5.5 
Backup type :  fs 

Error  :  
 "reason": "IndexShardSnapshotFailedException[Failed to snapshot]; nested: ElasticsearchException[failed to create blob container]; nested: AccessDeniedException[/backup/elasticsearch-backup/lb/indices/huUf9iZ9StCglJsOOsYs1A/0]; ",
          "node_id": "BYIubOVHR9G23ehNfEfZ7Q",
          "status": "INTERNAL_SERVER_ERROR"

Can  somebody  highlight or provide solution ,  why  Access Denied  exception is  comming  
Thanks

(David Pilato) #2

Please format your code using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

Please edit your post.

You probably have a stacktrace in logs. Could you share it ?


#3
stacktrace  

[2018-01-25T02:38:52,066][WARN ][o.e.s.SnapshotShardsService] [pc4cmelk02] [[.watcher-history-3-2017.12.26][0]] [lb_backup:lab_base1/pBNMVlaUSwGohP3nBsi0_g] failed to create snapshot
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: Failed to snapshot
        at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:397) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:88) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:335) [elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: org.elasticsearch.ElasticsearchException: failed to create blob container
        at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:70) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1078) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1071) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$SnapshotContext.<init>(BlobStoreRepository.java:1284) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:968) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:382) ~[elasticsearch-5.5.0.jar:5.5.0]
        ... 7 more
Caused by: java.nio.file.AccessDeniedException: /backup/elasticsearch-backup/lb/indices/u-zKaqTYSqWeJNe1vmBQDQ/0
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_101]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_101]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_101]
        at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:86) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:68) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1078) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.<init>(BlobStoreRepository.java:1071) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$SnapshotContext.<init>(BlobStoreRepository.java:1284) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:968) ~[elasticsearch-5.5.0.jar:5.5.0]
        at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:382) ~[elasticsearch-5.5.0.jar:5.5.0]

(David Pilato) #4
java.nio.file.AccessDeniedException: /backup/elasticsearch-backup/lb/indices/u-zKaqTYSqWeJNe1vmBQDQ/0

One of your nodes doesn't have access to this dir apparently?

When this error is appearing? What are the commands that you sent?


#5
Hi  
All the node have elasticsearch user permission to the repo path  "/backup/elasticsearch-backup/lb "
 while  taking  snaphot  from   "Cerebro v0.6.5"   it  creates  the  meta files  ,but get the Access denied error  while writing  to the "index "  folder 

Thanks 


(David Pilato) #6

Do you intentionally use this strange formatting for your text?

Could you ls -l your backup dir?

How many nodes you have?


#7

Hi
ls -l output

ls -l /backup/elasticsearch-backup/lb/

-rw-r--r--  1 488 488    29 Jan 23 09:20 incompatible-snapshots
-rw-r--r--  1 488 488    29 Jan 25 02:37 index-1
-rw-r--r--  1 488 488  5589 Jan 25 02:38 index-2
-rw-r--r--  1 488 488     8 Jan 25 02:38 index.latest
drwxr-xr-x 56 488 488  8192 Jan 25 02:38 indices
-rw-r--r--  1 488 488 15548 Jan 25 02:38 meta-pBNMVlaUSwGohP3nBsi0_g.dat
-rw-r--r--  1 488 488 19886 Jan 25 02:38 snap-pBNMVlaUSwGohP3nBsi0_g.dat

Thanks


(David Pilato) #8

How many nodes you have?


#9

Hi

5 nodes = 2 Data Node and 3 Master Node and X-Pack on top it .

Thanks
Himanshu


(David Pilato) #10

Is /backup/elasticsearch-backup/lb/ mount on every node and accessible via the user who is running the elasticsearch node locally.

Here I guess that one of the 5 nodes can not write there.


#11

Hi

All the node have same mount point and elasticsearch user have permision on it.

Thanks


(David Pilato) #12

Is elasticsearch user actually user 488 as I can see in the ls -l?


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.