# Trouble with errors when creating a snapshot **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [June 14, 2024, 6:01am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462 "2024-06-14T06:01:26Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 14, 2024, 6:01am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/1 "2024-06-14T06:01:26Z")
Hello from Japan I have a question for all the dear Elastic engineers I want to use the snapshot function to save a snapshot on one node in a cluster. \*I have configured a cluster of three Elasticsearch machines. I have introduced Samba to the server to store the snapshot repository. I am trying to create a shared folder with Samba and store the repository there. `However, many errors are bothering me.` ```auto Authentication status = Not connected { "name": "ResponseError", "message": "repository_verification_exception\n\tCaused by:\n\t\texception: failed to create blob container\n\tRoot causes:\n\t\texception: failed to create blob container" } ``` Error message when the shared folder source is set as the master. \*Server names and IP addresses are protected for security reasons. ```auto Authentication status = Not connected { "name": "ResponseError", "message": "repository_verification_exception\n\tRoot causes:\n\t\trepository_verification_exception: [repo_test] [[EPknCeSwRISYedwlqybijA, 'org.elasticsearch.transport.RemoteTransportException: [server name][xxx.xxx.xxx.xxx:xxxx][internal:admin/repository/verify]'], [-zN11OFCSPSBnvYOYu5kQg, 'org.elasticsearch.transport.RemoteTransportException: [server name][xxx.xxx.xxx.xxx:xxxx][internal:admin/repository/verify]']]" } ``` ☆My execution environment is as follows: > OS:Ubuntu 22.04 > Samba：4.15.13 > Elasticsearch：8.13.4 > Kibana：8.13.4 Each elasticsearch.yml file contains the repository settings as follows: > path.repo: /hayato Similar issues have been reported on the following sites, and I have investigated them but have been unable to resolve them. > > > For filesystem snapshots, if the 'elasticsearch' group and user have differing i…d numbers between nodes in the system, snapshots will not work. The master will create subdirectories for each index, but the permissions will reflect ownership by the 'elasticsearch' user on the master. If the master's elasticsearch user does not match the elasticsearch user id on another node, that node will not be able to write to those directories. > > When we create the 'elasticsearch' user and group at installation time, it would be helpful to have a consistent id number. I don't know if the best solution is to pick an arbitrary high user ID number (9200?) and try to use it by default, or just to document that it needs to be set the same across all nodes. I would appreciate any advice on the cause of the error, how to resolve it, and how to obtain a snapshot. 【The results of a survey I conducted on my own.】 ☆I suspected a problem with file system permissions, but I'm not sure of the details. ①The original permissions for the shared folder are displayed as drwxrwxrwx root root. When viewed from other mounted nodes, they are displayed as drwxr-xr-x root root, and cannot be changed with the chmod, chgrp, or chown commands. This may be why writing is not possible and the snapshot may be failing, but I don't know how to resolve it. ※I have confirmed that I can access the shared folder and write files from my own laptop. ②I checked the following site and found that the UID and GID of each node were different, so I changed them to the same values, but now elasticsearch.service won't start. > **[Shared file system repository | Elasticsearch Guide \[8.13\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/snapshots-filesystem-repository.html#filesystem-repository-settings)** ```auto Many NFS implementations match accounts across nodes using their numeric user IDs (UIDs) and group IDs (GIDs) rather than their names. It is possible for Elasticsearch to run under an account with the same name (often ) on each node, but for these accounts to have different numeric user or group IDs. If your shared file system uses NFS then ensure that every node is running with the same numeric UID and GID, or else update your NFS configuration to account for the variance in numeric IDs across nodes.elasticsearch ``` The Samba configuration file, smb.conf, is written as follows: ```auto [CCR] path = /hayato browsable = yes writable = yes guest ok = yes guest only = yes read only = no force create mode = 777 force directory mode = 777 ``` I am in a very difficult situation. Please, fellow Elasticsearch engineers, share your wisdom. Regards Thank you ※reference infomation The following is written in the Samba configuration file, smb.conf. /hayato is mounted as a shared folder on other nodes. > Filesystem Size Used Avail Use% Mounted on > tmpfs 794M 1.1M 793M 1% /run > /dev/vda1 1.9T 37G 1.9T 2% / > tmpfs 3.9G 28K 3.9G 1% /dev/shm > tmpfs 5.0M 0 5.0M 0% /run/lock > /dev/vda15 105M 6.1M 99M 6% /boot/efi > tmpfs 794M 4.0K 794M 1% /run/user/1010 > tmpfs 794M 4.0K 794M 1% /run/user/1001 > [//xxx.xxx.xxx.xxx/CCR](https://xxx.xxx.xxx.xxx/CCR) 1000G 57G 944G 6% /hayato ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 14, 2024, 7:27am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/2 "2024-06-14T07:27:39Z")
Hello @YUUTA.INOUE-JPN! To fix the problem we must first understand what's causing it. The error you've shared just says `failed to create blob container` but the full response from ES will include more details than just that simple message. There might also be relevant messages in the ES logs. You'll need to share more details. ---
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 16, 2024, 3:44pm UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/3 "2024-06-16T15:44:14Z")
good day @DavidTurner san \*In Japan, it is customary to add 【～san】 to the end of the name of someone you respect. Thank you for your reply. I was in such a difficult situation that I am very happy with your reply. I can provide you with Elasticsearch cluster logs. If there are any other logs you need, please let me know. I'm still new to Elasticsearch, so I don't understand everything that's needed yet. ```auto [2024-06-14T16:51:47,958][WARN][r.suppressed] [hostname] path: /_snapshot/repo_test/_verify, params: {repository=repo_ org.elasticsearch.transport.RemoteTransportException: [hostname][IP Adress:9300][cluster:admin/repository/verify] Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [repo_test] path is not accessible on master node Caused by: org.elasticsearch.ElasticsearchException: failed to create blob container at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:61) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2005) ~[elasticsearch- at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:499) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elast at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1583) ~[?:?] Caused by: java.nio.file.AccessDeniedException: /hayato/tests-Q0kMFYEgQYiGThx_n7HgPQ at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438) ~[?:?] at java.nio.file.Files.createDirectory(Files.java:699) ~[?:?] at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?] at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?] at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:59) ~[elasticsearch-8.13.4.jar:?] ``` ```auto failed to verify repository[2024-06-14T16:46:48,474][WARN][o.e.r.VerifyNodeRepositoryAction] [hostname] [repo_test] failed to veri org.elasticsearch.repositories.RepositoryVerificationException: [repo_test] store location [/hayato] is not accessible on the node [{xxxxxxxxx}{IP Adress}{IP Adress:9300}{cdfhilmrstw}{8.13.4}{7000099-8503000}{ml.allocated_processors_double=1.0, ml.max_jvm_size=107374182ersion=10.0.0, ml.machine_memory=2059472896, ml.allocated_processors=1}] Caused by: java.nio.file.AccessDeniedException: /hayato/tests-Euf88JLhSdiIfLc7hWJZZQ/data--zN11OFCSPSBnvYOYu5kQg.dat at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) ~[?:?] at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482) ~[?:?] at java.nio.file.Files.newOutputStream(Files.java:227) ~[?:?] ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2024, 4:19pm UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/4 "2024-06-16T16:19:19Z")
> [@YUUTA.INOUE-JPN](#): > > `org.elasticsearch.repositories.RepositoryVerificationException: [repo_test] path is not accessible on master node` The path must be accessible in the same location on all master and data nodes. The error message indicates it is not accessible from or configured on at least one of the master nodes. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 16, 2024, 6:23pm UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/5 "2024-06-16T18:23:00Z")
> [@Christian\_Dahlqvist](#): > > The path must be accessible in the same location on all master and data nodes. Well kinda but this is a very generic message, it could be caused by lots of things. The useful one is this one: > [@YUUTA.INOUE-JPN](#): > > ```auto > Caused by: java.nio.file.AccessDeniedException: /hayato/tests-Q0kMFYEgQYiGThx_n7HgPQ > > ``` That message indicates that the OS is forbidding Elasticsearch from creating the directory `/hayato/tests-Q0kMFYEgQYiGThx_n7HgPQ`. So there's something wrong with OS-level permissions. ---
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 17, 2024, 3:11am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/7 "2024-06-17T03:11:48Z")
@Christian_Dahlqvist san @DavidTurner san Thank you both for your replies. We will look into file system permissions. Currently we are using samba as the shared directory mechanism, and we think that the mechanism or parameters of samba may be affecting this. ※If you know of any websites or official documentation that introduces using samba to build an Elasticsearch snapshot mechanism, please let me know. Also, I'd like to research file system permissions and share what I find out. Please let me ask you questions again. Apart from this, we are facing another challenge: the ID of the Elasticsearch user that will create the repository for creating Elasticsearch snapshots. We know that to create a snapshot, we need an Elasticsearch account with the same permissions and user ID. However, when I try to change the user IDs in Ubuntu and align the user IDs of the three accounts, I am also facing the issue of not being able to start the Elasticsearch service. > > > For filesystem snapshots, if the 'elasticsearch' group and user have differing i…d numbers between nodes in the system, snapshots will not work. The master will create subdirectories for each index, but the permissions will reflect ownership by the 'elasticsearch' user on the master. If the master's elasticsearch user does not match the elasticsearch user id on another node, that node will not be able to write to those directories. > > When we create the 'elasticsearch' user and group at installation time, it would be helpful to have a consistent id number. I don't know if the best solution is to pick an arbitrary high user ID number (9200?) and try to use it by default, or just to document that it needs to be set the same across all nodes. These issues also exist as difficult issues on GitHub below. When I checked GitHub, I could not find a solution written there. Aside from changing the IDs of these Elasticsearch users at the OS level, how can I create a snapshot repository? I would appreciate any advice. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 17, 2024, 3:53am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/8 "2024-06-17T03:53:55Z")
> [@YUUTA.INOUE-JPN](#): > > Currently we are using samba as the shared directory mechanism, and we think that the mechanism or parameters of samba may be affecting this. > ※If you know of any websites or official documentation that introduces using samba to build an Elasticsearch snapshot mechanism, please let me know. I don't think we have official docs on this topic, sorry, it's more of a sysadmin question than anything directly related to Elasticsearch. > [@YUUTA.INOUE-JPN](#): > > We know that to create a snapshot, we need an Elasticsearch account with the same permissions and user ID. I don't think this is true, you just need to make sure that the user as which Elasticsearch is running on each node has the necessary permissions in your shared filesystem. I don't have any experience with doing this with Samba, nor do I see many other Samba experts around here. I think you'll get a more precise answer from your local sysadmin folks or a more Samba-related forum. ---
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 17, 2024, 4:35am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/9 "2024-06-17T04:35:14Z")
@DavidTurner san Thank you for your reply. I appreciate your feedback. > [@DavidTurner](#): > > I don't think this is true, you just need to make sure that the user as which Elasticsearch is running on each node has the necessary permissions in your shared filesystem. I'm a beginner with Elasticsearch, but I checked the official documentation and found the following: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a2520f469bfeeea91d76fa897414997d4652c449.png) > **[Shared file system repository | Elasticsearch Guide \[8.13\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/snapshots-filesystem-repository.html#filesystem-repository-settings)** The official documentation seems to require the same UID, but how should I understand this? I'm sorry that I don't understand Elasticsearch well. Regards Thank you ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 17, 2024, 4:51am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/10 "2024-06-17T04:51:14Z")
> [@YUUTA.INOUE-JPN](#): > > The official documentation seems to require the same UID, but how should I understand this? The docs you quote relate to NFS, not Samba, so I don't think they apply in your case. But also they don't say anything about requiring equal IDs - indeed they include a recommendation for how to deal with different IDs across nodes: > ensure that every node is running with the same numeric UID and GID, **or else update your NFS configuration to account for the variance in numeric IDs across nodes.** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 17, 2024, 4:54am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/11 "2024-06-17T04:54:14Z")
> [@YUUTA.INOUE-JPN](#): > > I'm a beginner with Elasticsearch 🙂 This isn't really an Elasticsearch problem, at least not now we've narrowed it down to the `AccessDeniedException` coming from the operating system. It's now just a case of fixing some filesystem permissions, but unfortunately this isn't a good place to ask for help about that. ---
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 18, 2024, 4:05am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/12 "2024-06-18T04:05:47Z")
@DavidTurner san Thank you for your reply. I would look into directory permissions first. If any new issues arise, I'd like to discuss this again. I'll reply here again when that happens. yuuta ---
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 18, 2024, 2:13pm UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/13 "2024-06-18T14:13:41Z")
@DavidTurner san and @Christian_Dahlqvist san Thanks to you, I was able to create a repository on the shared directory. The cause was indeed a filesystem permission issue. Your advice was very helpful. Thank you very much. However, when I tried to restore the cluster from the repository I created, the problem occurred again. When I tried to restore a snapshot, the following error message was displayed: ※ I perform these operations using the Kibana GUI. ※ These are the error messages displayed on the Kibana GUI. ```auto 20240618_TEST:hayato-elcnrb8nqh2gl97_c7dylg/xlSI8zSMTBuKtHPQWeqCoA] cannot restore index [.slo-observability.sli-v2] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name ``` I understand that this error message means that the snapshot restore failed because there are open indexes. I tried to close the indexes, but when I try to close the ".apm-agent-configuration" or ".tasks" indexes, for example, the following message is output and I cannot close them. ```auto action [indices:admin/close] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [apm-agent-configuration], this action is granted by the index privileges [manage_follow_index,manage,all] ``` ```auto action [indices:admin/close] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.tasks], this action is granted by the index privileges [manage_follow_index,manage,all］ ``` Also, the above index cannot be deleted. Could you please tell me how to solve this problem? First of all, how should we understand these indexes and data streams that cannot be CLOSE or deleted in terms of the purpose of creating and restoring snapshots? I would be grateful if you could let me know. Regards. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2024, 8:30am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/14 "2024-06-19T08:30:44Z")
Hmm I'm not sure I can help with this, but it's rather unrelated to your original message. I would suggest you start a new topic about it. ---
**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\ **Post date:** [June 19, 2024, 9:40am UTC](https://discuss.elastic.co/t/trouble-with-errors-when-creating-a-snapshot/361462/15 "2024-06-19T09:40:26Z")
@DavidTurner san thank you for your reply. I noted ! I'm very grateful for the support so far. Thank you！