java.io.IOException: could not remove the following files

Elastic Stack Elasticsearch
1 
**Describe the feature** : I created a backup repository to report an error。

**Elasticsearch version**  ( `bin/elasticsearch --version` ): 6.8.6

**JVM version**  ( `java -version` ): jdk-13.0.1+9

**OS version**  ( `uname -a`  if on a Unix-like system): centos 7  X86 64   kernel： 5.5.4-1.el7.elrepo.x86_64

I use the official docker image file “ docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.6 ” in the K8S。

1. I set the "path.repo" to /mnt and mounted the container's /mnt path to the local /mnt directory,
2. While I mounted the NFS Shared storage to the local /mnt directory.
3. I mount the same shared filesystem to the same location on all master and data nodes.

Here's what I did：

 curl -H 'Content-Type: application/json' -XPUT https://elasticsearch.root.com/_snapshot/2020-03-22 -d '
{
    "type": "fs",
    "settings": {
        "location": "/mnt/2020-03-22",
        "max_snapshot_bytes_per_sec": "50mb",
        "max_restore_bytes_per_sec": "50mb",
        "compress":true
    }
}'

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[2020-03-22] cannot delete test data at "}],"type":"repository_verification_exception","reason":"[2020-03-22] cannot delete test data at ","caused_by":{"type":"i_o_exception","reason":"could not remove the following files (in the order of attempts):\n /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg: java.nio.file.DirectoryNotEmptyException: /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg\n"}},"status":500}

The application reports an error message：

[2020-03-23T04:53:19,783][WARN ][r.suppressed             ] [es-cluster-elasticsearch-client-74c7b45d67-52l7m] path: /_snapshot/2020-03-22, params: {repository=2020-03-22}
org.elasticsearch.transport.RemoteTransportException: [es-cluster-elasticsearch-master-0][172.20.0.45:9300][cluster:admin/repository/put]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [2020-03-22] cannot delete test data at 
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:644) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:238) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.6.jar:6.8.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.io.IOException: could not remove the following files (in the order of attempts):
   /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg: java.nio.file.DirectoryNotEmptyException: /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg

	at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:197) ~[elasticsearch-core-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.delete(FsBlobStore.java:82) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:642) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:238) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.6.jar:6.8.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) ~[?:?]

[root@es-cluster-elasticsearch-data-0 mnt]# ll 2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg/
total 2
-rw-rw-r-- 1 elasticsearch root 22 Mar 23 04:53 data-UoFTkreDTLm4bJmDcXppfQ.dat
-rw-rw-r-- 1 elasticsearch root 22 Mar 23 04:53 data-sgmWV0csRUqteCDdXQQ5tw.dat
-rw-rw-r-- 1 elasticsearch root 22 Mar 23 04:53 data-ydD6f6WfRVylSItGi0go0w.dat
[root@es-cluster-elasticsearch-data-0 mnt]# ll -d /mnt/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 23 04:49 /mnt/
[root@es-cluster-elasticsearch-data-0 mnt]#
2

Welcome!

Wondering if you mounted your network dir as read only?

3

Thank you for your reply, it won't be read-only, because the directory 2020-03-22 was created after curl.

4

Do you meant that elasticsearch created the dir /mnt/2020-03-22?

5

yes, /mnt/2020-03-22 ,It's a directory created by elasticsearch.
Elasticsearch can successfully create directories and files in them, but it cannot delete them.

6

I believe it needs to be able to create as well as delete files.

7

You are right, but it can now be created but cannot be deleted. I have confirmed the system permission and found no problem

8

How many nodes do you have? Is it mounted read/write on ALL the nodes?

9

I have nine nodes, all of which can read and write to NFS Shared storage, and I just tested each one after you asked.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.