# Elasticsearch snapshot is failing due to access denied exception **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [January 5, 2024, 3:29pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474 "2024-01-05T15:29:08Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 5, 2024, 3:29pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/1 "2024-01-05T15:29:08Z")
Hi Team, Greetings ! A POC is under progress for the Elasticsearch snapshot and restoration and this POC is on the Azure VM's Linux environment. The version of Elasticsearch is 8.6.2 with three node cluster setup. The external storage setup is from the another Azure VM with 200 GB of disk space with in the same VLAN and exported that 200 GB mount point as a NFS storage as we don't have storage provision from the cloud. This repo is added to the elasticsearch.yml configuration file across all the nodes in the cluster. I have scheduled a SLM policy to take the snapshot on daily basis but it got failed due to "Failed to create blob container and access denied exception". At this point in time, full permission has been granted to the source and the destination mount points with user and group ownership as elasticsearch. Both SLM and manual snapshots are failed due to the access denied exceptions. Below are the log exceptions from the master node and can someone please help to resolve this issue? # few lines from elasticsearch.yml content path.data: /usr/share/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # path.repo: /etc/middleware action.destructive\_requires\_name: false # Exception from the logs: Suppressed: org.elasticsearch.ElasticsearchException: failed to create blob container at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:56) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.indexContainer(BlobStoreRepository.java:1630) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$46(BlobStoreRepository.java:1460) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.6.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?] Caused by: java.nio.file.AccessDeniedException: /etc/middleware/indices/8GVR\_UIGQ6WDcBklYUpcrw at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?] at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?] at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?] at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?] at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-8.6.2.jar:?] ... 8 more Caused by: java.nio.file.AccessDeniedException: /etc/middleware/indices/VSBpGoTdRa-uXxYyhyGj0g at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?] at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?] at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?] at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?] at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.indexContainer(BlobStoreRepository.java:1630) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$46(BlobStoreRepository.java:1460) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-8.6.2.jar:?] ... 5 more [2024-01-05T14:18:51,407][ERROR][o.e.x.s.SnapshotLifecycleTask] [gva6439] failed to create snapshot for snapshot lifecycle policy [backup\_policy1\_daily]: org.elasticsearch.snapshots.SnapshotException: [esbackup:backup-2024.01.05-nijkrpp9rd64cgwdk7q-1g/RRz3lwLmTUCSVyAQRt7M\_Q] failed to update snapshot in repository ---
**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\ **Post date:** [January 5, 2024, 4:17pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/2 "2024-01-05T16:17:48Z")
Hi, Verify the permissions on the /etc/middleware directory. Elasticsearch needs to have read, write, and execute permissions on this directory. You can check the permissions by running the command `ls -l /etc/middleware`. The output should show that the elasticsearch user has rwx permissions. Regards ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 5, 2024, 4:44pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/3 "2024-01-05T16:44:12Z")
Hi, Thanks for your response. File system /etc/middleware has full permission (777) with owner, group as elasticsearch. Regards, Karthick ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 5, 2024, 5:32pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/4 "2024-01-05T17:32:19Z")
Did you add the `path.repo` configuration before starting the nodes? What is the result of running on one of the nodes: ```auto $ sudo -u elasticsearch touch /etc/middleware/test-permission.txt ``` Also, any specific reason to use 8.6 when the last version is 8.11? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 6, 2024, 9:43am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/5 "2024-01-06T09:43:06Z")
> [@leandrojmp](#): > > Did you add the `path.repo` configuration before starting the nodes? I think you get a different error (mentioning `path.repo`) if that were the problem. > [@tykarthick](#): > > `java.nio.file.AccessDeniedException: /etc/middleware/indices/VSBpGoTdRa-uXxYyhyGj0g` Elasticsearch isn't doing anything magic here, it's just trying to create a directory as normal and this error is coming back from the OS. You will get the same problem if you try and run `mkdir /etc/middleware/indices/VSBpGoTdRa-uXxYyhyGj0g` from the command line as the user as which ES is running. ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 8, 2024, 9:01am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/6 "2024-01-08T09:01:03Z")
Hi Leandro Pereira, Thanks for your reply. Please find below the output. root @ gva6439 [QAS] /etc/middleware $ id uid=0(root) gid=0(root) groups=0(root) context=unconfined\_u:unconfined\_r:unconfined\_t:s0-s0:c0.c1023 root @ gva6439 [QAS] /etc/middleware $ sudo -u elasticsearch touch /etc/middleware/test-permission.txt root @ gva6439 [QAS] /etc/middleware $ ll -rw-r--r--. 1 elasticsearch elasticsearch 0 Jan 8 09:48 test-permission.txt root @ gva6439 [QAS] /etc/middleware $ id uid=0(root) gid=0(root) groups=0(root) context=unconfined\_u:unconfined\_r:unconfined\_t:s0-s0:c0.c1023 root @ gva6439 [QAS] /etc/middleware $ sudo -u elasticsearch usage: sudo -h | -K | -k | -V usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user] usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user] [command] usage: sudo [-AbEHknPS] [-r role] [-t type] [-C num] [-g group] [-h host] [-p prompt] [-T timeout] [-u user] [VAR=value] [-i|-s] [] usage: sudo -e [-AknS] [-r role] [-t type] [-C num] [-g group] [-h host] [-p prompt] [-T timeout] [-u user] file ... root @ gva6439 [QAS] /etc/middleware $ su - elasticsearch su: warning: cannot change directory to /nonexistent: No such file or directory This account is currently not available. root @ gva6439 [QAS] /etc/middleware $ Our live environments are running on 8.6 and hence we wanted to perform the snapshot and restoration POC on 8.6 and hence no testing's on the latest version 8.11. Please let me know how to proceed further to resolve this issue. ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 8, 2024, 9:01am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/7 "2024-01-08T09:01:51Z")
Hi David, Thanks for your reply. Yes, path.repo was added before starting the nodes. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 8, 2024, 10:03am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/8 "2024-01-08T10:03:34Z")
> [@tykarthick](#): > > ```auto > $ id > uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > > ``` Oh, you're using SELinux. It would have been worth mentioning that up front! That makes it all a lot more complicated, but the problem lies in your SELinux config and not in Elasticsearch so you will need to work with your local sysadmin folks to fix it. As I said, Elasticsearch isn't doing anything magic, it's just trying to create a directory. ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 8, 2024, 11:44am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/9 "2024-01-08T11:44:03Z")
Hi David, Tried all possibilities from OS perspective but nothing worked. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 8, 2024, 11:58am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/10 "2024-01-08T11:58:04Z")
Sorry, there's nothing we can do on the Elasticsearch side to fix this. You need someone with SELinux expertise. ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 8, 2024, 1:44pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/11 "2024-01-08T13:44:04Z")
Thanks for the update, David. ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 12, 2024, 1:26pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/12 "2024-01-12T13:26:28Z")
Hi Leandro, The issue is still open and any further update from your side is highly appreciable? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 12, 2024, 2:13pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/13 "2024-01-12T14:13:42Z")
Hello, I have no update on this, as David mentioned this does not seem an issue with Elasticsearch, but to the underlying permissions on your system and SELinux. I do not have much experience with SELinux, so I can not help further. ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 12, 2024, 2:47pm UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/14 "2024-01-12T14:47:40Z")
Hi Leandro, Thanks for the update. ---
**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\ **Post date:** [January 14, 2024, 9:39am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/15 "2024-01-14T09:39:42Z")
Hi, If SELinux is enabled on your system, you can use the `chcon` command to change the security context of the `/etc/middleware` directory to allow Elasticsearch to write to it. Here's an example: ```auto sudo chcon -Rv --type=elasticsearch_var_lib_t /etc/middleware ``` **Note that it's only a logical thought** derived from reading this link: [Grant privileged permissions to Beats | Elastic Cloud on Kubernetes [1.3] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/1.3/k8s-openshift-beats.html)" Regards ---
**Author:** ![tykarthick](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@tykarthick](https://discuss.elastic.co/u/tykarthick)\ **Post date:** [January 16, 2024, 9:28am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/16 "2024-01-16T09:28:25Z")
Hi Yago, Thanks for your reply. I have compared the ls -lZ /etc/middleware output with the working environment (lab servers - two node elasticsearch cluster) along with Azure DR servers where we have the snapshot issue and the output of the commands furnished below for your reference. It seems that the permission are not different for both working and non working environment. In this case, please let me know what needs to be changed on the non working environment. Snapshot and restoration is working fine as expected on the lab environment and output of ls -lZ /etc/middleware [SBX] ~ $ ls -lZ /etc/middleware/ total 112 drwxrwxrwx. 3 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 4096 Oct 27 12:30 backup -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 9083 Jan 16 02:30 index-152 -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 8 Jan 16 02:30 index.latest drwxr-xr-x. 19 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 4096 Dec 17 13:32 indices -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 19312 Jan 13 13:32 meta-6WzpVUYuQKWQdz-PSklMaQ.dat -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 19313 Jan 14 13:32 meta-o69tAELBS-WuTk0CkuGOQQ.dat -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 19311 Jan 15 13:32 meta-vRvG7LJSTtSJGJ-pd-vwvg.dat -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 604 Jan 13 13:32 snap-6WzpVUYuQKWQdz-PSklMaQ.dat -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 601 Jan 14 13:32 snap-o69tAELBS-WuTk0CkuGOQQ.dat -rw-r--r--. 1 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 599 Jan 15 13:32 snap-vRvG7LJSTtSJGJ-pd-vwvg.dat drwxr-xr-x. 2 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 4096 Oct 27 13:47 test drwxrwxrwx. 3 768 768 system\_u:object\_r:nfs\_t:s0 4096 Oct 27 12:51 testrepo drwxr-xr-x. 2 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 4096 Nov 2 09:15 tests-gptAluNTTgSmOuD\_o1kq1Q drwxr-xr-x. 2 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 4096 Dec 12 13:44 tests-LLUbX7UeTWGgFAJoPsFsFw drwxr-xr-x. 2 elasticsearch elasticsearch system\_u:object\_r:nfs\_t:s0 4096 Oct 27 12:59 tests-wIF1pmNeT0maTfbnvrQ-aA -rwxrwxrwx. 1 root root system\_u:object\_r:nfs\_t:s0 0 Nov 1 12:05 test.txt Output from Azure DR servers where the snapshot is not working as expected: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/3/33f58cfe0c6ea8d20d2273abb4f3cab961e0d62e.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 13, 2024, 9:29am UTC](https://discuss.elastic.co/t/elasticsearch-snapshot-is-failing-due-to-access-denied-exception/350474/17 "2024-02-13T09:29:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.