Hi all,
I'm trying to snapshot an index to an fs repository (nfs share).
I have two nodes in 7.6.1. First node share nfs, second mount it
There is around 80 indices (30 GB/day), I decide to snapshot monthly manually before setting an ILM that will delete cold index after 30 days. I need to keep backup for 1 year
First month run smoothly and snapshot 18 index, other dosen't and report error bellow in police history
{
"type": "snapshot_exception",
"reason": "[faz_archives:snapshot-faz-fj2wco52rh2b-g0dgjq1hw/CCvUmPNxTfSQe9mQKyphbQ] failed to update snapshot in repository",
"caused_by": {
"type": "file_system_exception",
"reason": "/var/archives/pending-index.latest-g7Kq4PgbTtq2-Uz1JW3B6A: Structure needs cleaning",
"stack_trace": "java.nio.file.FileSystemException: /var/archives/pending-index.latest-g7Kq4PgbTtq2-Uz1JW3B6A: Structure needs cleaning\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\n\tat sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)\n\tat java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)\n\tat java.nio.file.Files.newOutputStream(Files.java:216)\n\tat org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlobAtomic(FsBlobContainer.java:174)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1455)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$47(BlobStoreRepository.java:1335)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$45(BlobStoreRepository.java:1312)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"
},
...
}
There are folders created in repository, but I don't see my snapshots.
Also, Data drive is growing each time I launch snapshot policy !
How can I clean structure as requested and recover data drive space ?
Thanks for help