Unable to create snapshots with cifs share 9.4.0

Elastic Stack Elasticsearch
1

Hello,

Since upgrading to 9.4.0, we are experiencing snapshot failures on our CIFS-mounted shared repository.

Timeline:

  • Last successful snapshot: May 6, before the upgrade
  • Upgrade to 9.4.0 performed: May 6 at 12:00 PM Europe/Paris
  • All snapshot attempts have failed since then

Repository configuration:

json

{
  "elasticsearch": {
    "type": "fs",
    "uuid": "46diYBCrQLqmDchEbLpwVQ",
    "settings": {
      "location": "/svg/backup"
    }
  }
}

/svg/backup is a CIFS mount present on all Elasticsearch nodes, and path.repo is defined in elasticsearch.yml on each node.

Snapshot failed log

{
  "type": "snapshot_exception",
  "reason": "[elasticsearch:2026.05.11-daily-snapshot-elasticsearch-lk-qkzvyshglta6cz0xjig/tq5pklx9TkepDZ-HdphGLA] failed to update snapshot in repository",
  "caused_by": {
    "type": "file_system_exception",
    "reason": "/svg/backup/index-2453 -> /svg/backup/pending-index-2453-KXOuCfGoQvaWU-SI_n_pHw: Opération non supportée"
  }
}

Comments

Creating/moving files in the share from elastic host with elasticsearch user work.
Creating hardlink don't.

I don't know the way elastic performed snapshot before 9.4.0, but it seems to use hardlink now. Which is not well supported in CIFS.

I didn't see any informations about snapshots in breaking change section of the release note.

Questions:

  • Is this an intentional change or a regression?

Logs:

Repository verification:

org.elasticsearch.repositories.RepositoryVerificationException: [elasticsearch] path  is not accessible on master node
Caused by: java.nio.file.FileSystemException: /svg/backup/tests-mXWL-4lfTNKJI-PdsCUdxw/master.dat -> /svg/backup/tests-mXWL-4lfTNKJI-PdsCUdxw/pending-master.dat-bg4uXHJSQ4GOLBDkKhswvA: Opération non supportée
   at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:102)
   at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:108)
   at java.base/sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:528)
   at java.base/java.nio.file.Files.createLink(Files.java:1020)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.blobstore.fs.FsBlobContainer.moveAtomicallyUsingHardLink(FsBlobContainer.java:430)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.blobstore.fs.FsBlobContainer.moveBlobAtomic(FsBlobContainer.java:408)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlobAtomic(FsBlobContainer.java:343)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2477)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:697)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1114)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
   at java.base/java.lang.Thread.run(Thread.java:1516)

Snapshot failed logs:


failed to create snapshot for snapshot lifecycle policy [daily-snapshot]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[LXELK01][masterService#updateTask][T#1427]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","trace.id":"bbeb59e48798085598552b3022b697b8","elasticsearch.cluster.uuid":"z0kseJw_QUu8UQMoRXNsmg","elasticsearch.node.id":"fj8fXLCSQOWEYhQ4Qpwa2Q","elasticsearch.node.name":"LXELK01","elasticsearch.cluster.name":"es-cluster","error.type":"org.elasticsearch.snapshots.SnapshotException","error.message":"[elasticsearch:2026.05.11-daily-snapshot-elasticsearch-lk-qkzvyshglta6cz0xjig/tq5pklx9TkepDZ-HdphGLA] failed to update snapshot in repository","error.stack_trace":"org.elasticsearch.snapshots.SnapshotException: [elasticsearch:2026.05.11-daily-snapshot-elasticsearch-lk-qkzvyshglta6cz0xjig/tq5pklx9TkepDZ-HdphGLA] failed to update snapshot in repository
   at org.elasticsearch.server@9.4.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$31(BlobStoreRepository.java:2089)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:448)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:351)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:376)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:288)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:448)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:351)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:376)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:288)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:263)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:436)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:347)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:376)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:283)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$47(BlobStoreRepository.java:3123)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1114)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
   at java.base/java.lang.Thread.run(Thread.java:1516)
Caused by: java.nio.file.FileSystemException: /svg/backup/index-2453 -> /svg/backup/pending-index-2453-KXOuCfGoQvaWU-SI_n_pHw: Opération non supportée
   at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:102)
   at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:108)
   at java.base/sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:528)
   at java.base/java.nio.file.Files.createLink(Files.java:1020)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.blobstore.fs.FsBlobContainer.moveAtomicallyUsingHardLink(FsBlobContainer.java:430)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.blobstore.fs.FsBlobContainer.moveBlobAtomic(FsBlobContainer.java:408)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeMetadataBlob(FsBlobContainer.java:271)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:3462)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$50(BlobStoreRepository.java:3146)
   at org.elasticsearch.server@9.4.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
   ... 11 more

Edit 11/05/2026 15:13:

I tried with es-start-local version 9.3.3. And snapshot works without issues
with es-start-local version 9.4.0, we experiencing same issue described above.

2

Thank you @expErg for sharing this.

The hard link approach for moving files has recently been introduced to Elasticsearch to solve a problem of a race condition in this area.

There is a fallback for non-compliant systems, but apparently a different exception than anticipated is thrown there (we catch UnsupportedOperationException, as defined in java.nio.file.Files#createLink but it looks like the the UnixFileSystemProvider throws a different one).

3

Thank you @michalborek for the valuable information.
After a little more research, I was able to find the PR that introduced this behavior.

Should I open an issue on GitHub to track this ?

4

If you open an issue, I can link it with my PR (we'd have your installation details that may help in future cases).

I proposed a PR with the fix here: Fix FsBlobContainer.moveBlobAtomic fallback on CIFS filesystems by michalborek · Pull Request #148777 · elastic/elasticsearch · GitHub .

5

The issue.
Thank you for the quick response!