Regarding the error message when clicking the Validate Repository button in Elasticsearch snapshot function

Elastic Stack Elasticsearch
1

Hello from Japan
I have a question for all the dear Elastic engineers
I want to use the snapshot function to save a snapshot on one node in a cluster.
*I have configured a cluster of three Elasticsearch machines.

I have introduced Samba to the server to store the snapshot repository.
I am trying to create a shared folder with Samba and store the repository there.

My Elastic environment is as follows:

OS:Ubuntu 22.04
Samba：4.15.13
Elasticsearch：8.13.4
Kibana：8.13.4

I was able to solve various errors with the help of the official documentation and Elastic Discuss, but I am currently working on solving the last problem.
This is about an error that occurs when setting up a snapshot repository via the Kibana GUI.
When I clicked "Verify Repository" in the "Authentication Status" section of the repository I created, the following error was output.

{
  "name": "ResponseError",
  "message": "repository_verification_exception\n\tCaused by:\n\t\tdirectory_not_empty_exception: /opt/snapshot/tests-ilf9yeixQl26SgWmb2qJ3w\n\tRoot causes:\n\t\trepository_verification_exception: [20240718_backup_server-hostname] cannot delete test data at "
}

When I checked the Elastic cluster log, the following log was output:

        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2019) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.RepositoriesService$4.lambda$doRun$1(RepositoriesService.java:508) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-07-18T14:39:40,399][WARN ][o.e.r.RepositoriesService] [backup_server-hostname] [20240718_backup_server-hostname] failed to finish repository verification
org.elasticsearch.repositories.RepositoryVerificationException: [20240718_backup_server-hostname] cannot delete test data at
Caused by: java.nio.file.DirectoryNotEmptyException: /opt/snapshot/tests-HATebPVrSBCmmXAd2yYsxw
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:289) ~[?:?]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:104) ~[?:?]
        at java.nio.file.Files.delete(Files.java:1152) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:159) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2019) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.RepositoriesService$4.lambda$doRun$1(RepositoriesService.java:508) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-07-18T14:39:40,401][WARN ][r.suppressed             ] [backup_server-hostname] path: /_snapshot/20240718_backup_server-hostname/_verify, params: {repository=20240718_backup_server-hostname}, status: 500
org.elasticsearch.repositories.RepositoryVerificationException: [20240718_backup_server-hostname] cannot delete test data at
Caused by: java.nio.file.DirectoryNotEmptyException: /opt/snapshot/tests-HATebPVrSBCmmXAd2yYsxw
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:289) ~[?:?]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:104) ~[?:?]
        at java.nio.file.Files.delete(Files.java:1152) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:159) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer$2.postVisitDirectory(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:155) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2019) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.repositories.RepositoriesService$4.lambda$doRun$1(RepositoriesService.java:508) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.13.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

Even though the above error is output, the snapshot appears to be taken successfully.
I would appreciate your help in finding out what this error means, whether there is a problem with the snapshot that is taken, and what the solution is.

[2024-07-18T14:47:01,751][INFO ][o.e.x.s.SnapshotLifecycleTask] [backup_server-hostname] snapshot lifecycle policy [monthly] issuing create snapshot [monthly-2z100                  a2ptom2--ib2c5r-g]
[2024-07-18T14:47:01,950][INFO ][o.e.s.SnapshotsService   ] [backup_server-hostname] snapshot [20240718_backup_server-hostname:monthly-2z100a2ptom2--ib2c5r-g/lWCqXFBxQlOlYzev                  0rabTA] started
[2024-07-18T14:47:04,754][INFO ][o.e.s.SnapshotsService   ] [backup_server-hostname] snapshot [20240718_backup_server-hostname:monthly-2z100a2ptom2--ib2c5r-g/lWCqXFBxQlOlYzev                  0rabTA] completed with state [SUCCESS]

I have found similar situations on Elastic Discuss: Elasticsearch Cluster Multi Node Shared File System Repository SMB Issue
I checked, but unfortunately it seems that the above ticket did not lead to a solution.

I'd love to hear from Elastic engineers sharing their wisdom.
Thank you

2

Hi again @YUUTA.INOUE-JPN :wave:

This error indicates that your Samba installation isn't behaving correctly - Elasticsearch is creating a directory, adding some files, deleting those files again, but then when it tries to delete the directory the OS returns an error saying that the directory isn't empty. But it must be empty, we just successfully deleted all the files in it! Unfortunately I don't know how to solve this, it's not really an Elasticsearch issue so much as a Samba one.

3

@DavidTurner san :grinning:
I'm happy to be able to talk to you again.
I am always happy to receive advice and it helps me a lot.

I will follow your advice and try reinstalling samba.
Also, if you have any other good ideas, please let me know.

Additionally, I have one more question.
Looking at the log, it seems that the repository verification was not completed, but I wonder if there are any problems with the operation of the snapshot itself?
※I'm not very familiar with Elasticsearch, so I'd like to get positive proof.

4

If verification fails then we cannot say whether snapshots will work correctly or not. It may be that they mostly work but I doubt that is satisfactory.

See these docs for more information.

5

@YUUTA.INOUE-JPN were you able to figure this out?