# java.io.IOException: could not remove the following files **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [March 23, 2020, 5:11am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591 "2020-03-23T05:11:50Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![xlogin](https://avatars.discourse-cdn.com/v4/letter/x/cdc98d/32.png) [@xlogin](https://discuss.elastic.co/u/xlogin)\ **Post date:** [March 23, 2020, 5:11am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/1 "2020-03-23T05:11:50Z")
```auto **Describe the feature** : I created a backup repository to report an error。 **Elasticsearch version** ( `bin/elasticsearch --version` ): 6.8.6 **JVM version** ( `java -version` ): jdk-13.0.1+9 **OS version** ( `uname -a` if on a Unix-like system): centos 7 X86 64 kernel： 5.5.4-1.el7.elrepo.x86_64 ``` ```auto I use the official docker image file “ docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.6 ” in the K8S。 1. I set the "path.repo" to /mnt and mounted the container's /mnt path to the local /mnt directory, 2. While I mounted the NFS Shared storage to the local /mnt directory. 3. I mount the same shared filesystem to the same location on all master and data nodes. ``` Here's what I did： ```auto curl -H 'Content-Type: application/json' -XPUT https://elasticsearch.root.com/_snapshot/2020-03-22 -d ' { "type": "fs", "settings": { "location": "/mnt/2020-03-22", "max_snapshot_bytes_per_sec": "50mb", "max_restore_bytes_per_sec": "50mb", "compress":true } }' ``` `{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[2020-03-22] cannot delete test data at "}],"type":"repository_verification_exception","reason":"[2020-03-22] cannot delete test data at ","caused_by":{"type":"i_o_exception","reason":"could not remove the following files (in the order of attempts):\n /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg: java.nio.file.DirectoryNotEmptyException: /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg\n"}},"status":500}` The application reports an error message： ```auto [2020-03-23T04:53:19,783][WARN][r.suppressed] [es-cluster-elasticsearch-client-74c7b45d67-52l7m] path: /_snapshot/2020-03-22, params: {repository=2020-03-22} org.elasticsearch.transport.RemoteTransportException: [es-cluster-elasticsearch-master-0][172.20.0.45:9300][cluster:admin/repository/put] Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [2020-03-22] cannot delete test data at at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:644) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:238) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.6.jar:6.8.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:830) [?:?] Caused by: java.io.IOException: could not remove the following files (in the order of attempts): /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg: java.nio.file.DirectoryNotEmptyException: /mnt/2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:197) ~[elasticsearch-core-6.8.6.jar:6.8.6] at org.elasticsearch.common.blobstore.fs.FsBlobStore.delete(FsBlobStore.java:82) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:642) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:238) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.6.jar:6.8.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:830) ~[?:?] ``` ```auto [root@es-cluster-elasticsearch-data-0 mnt]# ll 2020-03-22/tests-bROfi9MOQOy8Xr4RifViWg/ total 2 -rw-rw-r-- 1 elasticsearch root 22 Mar 23 04:53 data-UoFTkreDTLm4bJmDcXppfQ.dat -rw-rw-r-- 1 elasticsearch root 22 Mar 23 04:53 data-sgmWV0csRUqteCDdXQQ5tw.dat -rw-rw-r-- 1 elasticsearch root 22 Mar 23 04:53 data-ydD6f6WfRVylSItGi0go0w.dat [root@es-cluster-elasticsearch-data-0 mnt]# ll -d /mnt/ drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 23 04:49 /mnt/ [root@es-cluster-elasticsearch-data-0 mnt]# ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 23, 2020, 8:20am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/2 "2020-03-23T08:20:07Z")
Welcome! Wondering if you mounted your network dir as read only? ---
**Author:** ![xlogin](https://avatars.discourse-cdn.com/v4/letter/x/cdc98d/32.png) [@xlogin](https://discuss.elastic.co/u/xlogin)\ **Post date:** [March 23, 2020, 9:50am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/3 "2020-03-23T09:50:43Z")
Thank you for your reply, it won't be read-only, because the directory 2020-03-22 was created after curl. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 23, 2020, 10:09am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/4 "2020-03-23T10:09:07Z")
Do you meant that elasticsearch created the dir `/mnt/2020-03-22`? ---
**Author:** ![xlogin](https://avatars.discourse-cdn.com/v4/letter/x/cdc98d/32.png) [@xlogin](https://discuss.elastic.co/u/xlogin)\ **Post date:** [March 23, 2020, 10:36am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/5 "2020-03-23T10:36:02Z")
yes, /mnt/2020-03-22 ,It's a directory created by elasticsearch. Elasticsearch can successfully create directories and files in them, but it cannot delete them. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2020, 10:45am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/6 "2020-03-23T10:45:37Z")
I believe it needs to be able to create as well as delete files. ---
**Author:** ![xlogin](https://avatars.discourse-cdn.com/v4/letter/x/cdc98d/32.png) [@xlogin](https://discuss.elastic.co/u/xlogin)\ **Post date:** [March 23, 2020, 10:48am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/7 "2020-03-23T10:48:06Z")
You are right, but it can now be created but cannot be deleted. I have confirmed the system permission and found no problem ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2020, 10:49am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/8 "2020-03-23T10:49:24Z")
How many nodes do you have? Is it mounted read/write on ALL the nodes? ---
**Author:** ![xlogin](https://avatars.discourse-cdn.com/v4/letter/x/cdc98d/32.png) [@xlogin](https://discuss.elastic.co/u/xlogin)\ **Post date:** [March 23, 2020, 11:02am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/9 "2020-03-23T11:02:38Z")
I have nine nodes, all of which can read and write to NFS Shared storage, and I just tested each one after you asked. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2020, 11:02am UTC](https://discuss.elastic.co/t/java-io-ioexception-could-not-remove-the-following-files/224591/10 "2020-04-20T11:02:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.