# Regarding the error message when clicking the Validate Repository button in Elasticsearch snapshot function

**URL:** <https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [July 18, 2024, 8:37am UTC](https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333 "2024-07-18T08:37:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\
**Post date:** [July 18, 2024, 8:37am UTC](https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333/1 "2024-07-18T08:37:33Z")

</div>

Hello from Japan  
I have a question for all the dear Elastic engineers  
I want to use the snapshot function to save a snapshot on one node in a cluster.  
\*I have configured a cluster of three Elasticsearch machines.

I have introduced Samba to the server to store the snapshot repository.  
I am trying to create a shared folder with Samba and store the repository there.

My Elastic environment is as follows:

> OS:Ubuntu 22.04  
> Samba：4.15.13  
> Elasticsearch：8.13.4  
> Kibana：8.13.4

I was able to solve various errors with the help of the official documentation and Elastic Discuss, but I am currently working on solving the last problem.  
This is about an error that occurs when setting up a snapshot repository via the Kibana GUI.  
When I clicked "Verify Repository" in the "Authentication Status" section of the repository I created, the following error was output.

```auto
{
  "name": "ResponseError",
  "message": "repository_verification_exception\n\tCaused by:\n\t\tdirectory_not_empty_exception: /opt/snapshot/tests-ilf9yeixQl26SgWmb2qJ3w\n\tRoot causes:\n\t\trepository_verification_exception: [20240718_backup_server-hostname] cannot delete test data at "
}

```

When I checked the Elastic cluster log, the following log was output:

```auto
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2019) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.RepositoriesService$4.lambda$doRun$1(RepositoriesService.java:508) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-07-18T14:39:40,399][WARN][o.e.r.RepositoriesService] [backup_server-hostname] [20240718_backup_server-hostname] failed to finish repository verification
org.elasticsearch.repositories.RepositoryVerificationException: [20240718_backup_server-hostname] cannot delete test data at
Caused by: java.nio.file.DirectoryNotEmptyException: /opt/snapshot/tests-HATebPVrSBCmmXAd2yYsxw
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:289) ~[?:?]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:104) ~[?:?]
        at java.nio.file.Files.delete(Files.java:1152) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:159) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2019) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.RepositoriesService$4.lambda$doRun$1(RepositoriesService.java:508) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-07-18T14:39:40,401][WARN][r.suppressed] [backup_server-hostname] path: /_snapshot/20240718_backup_server-hostname/_verify, params: {repository=20240718_backup_server-hostname}, status: 500
org.elasticsearch.repositories.RepositoryVerificationException: [20240718_backup_server-hostname] cannot delete test data at
Caused by: java.nio.file.DirectoryNotEmptyException: /opt/snapshot/tests-HATebPVrSBCmmXAd2yYsxw
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:289) ~[?:?]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:104) ~[?:?]
        at java.nio.file.Files.delete(Files.java:1152) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:159) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2019) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.RepositoriesService$4.lambda$doRun$1(RepositoriesService.java:508) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

```

Even though the above error is output, the snapshot appears to be taken successfully.  
I would appreciate your help in finding out what this error means, whether there is a problem with the snapshot that is taken, and what the solution is.

```auto
[2024-07-18T14:47:01,751][INFO][o.e.x.s.SnapshotLifecycleTask] [backup_server-hostname] snapshot lifecycle policy [monthly] issuing create snapshot [monthly-2z100 a2ptom2--ib2c5r-g]
[2024-07-18T14:47:01,950][INFO][o.e.s.SnapshotsService] [backup_server-hostname] snapshot [20240718_backup_server-hostname:monthly-2z100a2ptom2--ib2c5r-g/lWCqXFBxQlOlYzev 0rabTA] started
[2024-07-18T14:47:04,754][INFO][o.e.s.SnapshotsService] [backup_server-hostname] snapshot [20240718_backup_server-hostname:monthly-2z100a2ptom2--ib2c5r-g/lWCqXFBxQlOlYzev 0rabTA] completed with state [SUCCESS]

```

I have found similar situations on Elastic Discuss: [Elasticsearch Cluster Multi Node Shared File System Repository SMB Issue](https://discuss.elastic.co/t/elasticsearch-cluster-multi-node-shared-file-system-repository-smb-issue/328472)  
I checked, but unfortunately it seems that the above ticket did not lead to a solution.

I'd love to hear from Elastic engineers sharing their wisdom.  
Thank you

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 18, 2024, 12:12pm UTC](https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333/2 "2024-07-18T12:12:41Z")

</div>

Hi again @YUUTA.INOUE-JPN 👋

This error indicates that your Samba installation isn't behaving correctly - Elasticsearch is creating a directory, adding some files, deleting those files again, but then when it tries to delete the directory the OS returns an error saying that the directory isn't empty. But it must be empty, we just successfully deleted all the files in it! Unfortunately I don't know how to solve this, it's not really an Elasticsearch issue so much as a Samba one.

---

<div class="post-metadata">

**Author:** ![YUUTA.INOUE-JPN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuuta.inoue-jpn/32/117963_2.png) [@YUUTA.INOUE-JPN](https://discuss.elastic.co/u/YUUTA.INOUE-JPN)\
**Post date:** [July 19, 2024, 12:56am UTC](https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333/3 "2024-07-19T00:56:11Z")

</div>

@DavidTurner san 😀  
I'm happy to be able to talk to you again.  
I am always happy to receive advice and it helps me a lot.

I will follow your advice and try reinstalling samba.  
Also, if you have any other good ideas, please let me know.

Additionally, I have one more question.  
Looking at the log, it seems that the repository verification was not completed, but I wonder if there are any problems with the operation of the snapshot itself?  
※I'm not very familiar with Elasticsearch, so I'd like to get positive proof.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 19, 2024, 3:46am UTC](https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333/4 "2024-07-19T03:46:11Z")

</div>

If verification fails then we cannot say whether snapshots will work correctly or not. It may be that they _mostly_ work but I doubt that is satisfactory.

See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-filesystem-repository.html#_troubleshooting_a_shared_file_system_repository) for more information.

---

<div class="post-metadata">

**Author:** ![damian\_harouff](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damian_harouff/32/136758_2.png) [@damian\_harouff](https://discuss.elastic.co/u/damian_harouff)\
**Post date:** [August 13, 2024, 6:11pm UTC](https://discuss.elastic.co/t/regarding-the-error-message-when-clicking-the-validate-repository-button-in-elasticsearch-snapshot-function/363333/5 "2024-08-13T18:11:14Z")

</div>

@YUUTA.INOUE-JPN were you able to figure this out?
