java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs

Elastic Stack Elasticsearch
1

Hi,

I have a problem with one of my elasticsearch node. For some reason node was shutdown due to some error. When I look into the log, I get the error java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs, see full error below.

{
  "@timestamp": "2026-09-08T09:38:55.683Z",
  "log.level": "ERROR",
  "message": "fatal exception while booting Elasticsearch",
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.server",
  "process.thread.name": "main",
  "log.logger": "org.elasticsearch.bootstrap.Elasticsearch",
  "elasticsearch.node.name": "es03-hot",
  "elasticsearch.cluster.name": "s1em",
  "error.type": "org.elasticsearch.ElasticsearchException",
  "error.message": "Failed to bind service",
  "error.stack_trace": "org.elasticsearch.ElasticsearchException: Failed to bind service\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:276)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.Node.<init>(Node.java:192)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)\nCaused by: org.apache.lucene.index.CorruptIndexException: Problem reading index. (resource=/usr/share/elasticsearch/data/_state/_pu2t.cfs)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:167)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:94)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:820)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:353)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:611)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:334)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:500)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)\n\t... 4 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:224)\n\tat java.base/java.nio.channels.FileChannel.open(FileChannel.java:309)\n\tat java.base/java.nio.channels.FileChannel.open(FileChannel.java:369)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:78)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:87)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)\n\t... 16 more\n"
}

When I look into /usr/share/elasticsearch/data/_state/ catalog, the file is actually missing.

How can I resolve that error?

Thanks in advance.

2

Hi @TheJ,

Can you share which version of Elasticsearch you are using? Have you recently upgraded? Do you have any background processes running that are deleting any files?

Let us know!

3

Thank you for quick response.

My elasticsearch cluster is in 8.13.0 version. I check for other processes on that server and there is only elasticsearch with docker so this isn't possible to delete that file by some process. Also, I'm 100% sure that file has not been deleted manually.

4

Can you share how you are starting your elasticsearch container?

5

Does this file exist on other nodes? [Please do not copy manually _pu2t.cfs]

6

Sure, I have docker compose configuration file and this cluster is deployed with docker stack deploy --compose-file compose.yml s1em command.

Also I have mounted volume for data on host machine (not as docker volume).

7

Nope, other nodes work without any problems and they are don't have that file

8

Please share your compose.yml file and other relevant configuration files.

9

Yeah, the specific file name is not that important here, as it would be a different something.cfs filename on other nodes in your cluster. More is that elasticsearch expects to find that file and does not find it. Is there anything in _state director on that node?

Yeah, the full error and sequence that led to node crash is likely more interesting. Including timestamps. "fatal exception while booting Elasticsearch" tells us it wont boot, I want to know why it crashed.

Sorry, experience tells me not to share your "isn't possible" confidence. It's possible, maybe unlikely, TBD.

10

100% this.

The only way to get Elasticsearch into a state where it needs to read this file is if it previously created this file and it definitely hasn't deleted it. The creation of this file included successful responses from fsync() indicating that this operation had become durable. The non-deletion of this file is because Elasticsearch would first have performed equally-durable writes indicating the file is no longer needed.

It is overwhelmingly likely that the explanation is that something on your system deleted this file out from underneath Elasticsearch. It may not be a process which is still running, so checking for other processes on the same machine is not conclusive.

One other possibility is that your storage subsystem reports success from writes and fsync() operations before those writes have become durable, which can become problematic if there were a power outage or other hardware failure at the wrong moment. But you didn't mention such a failure so it's probably not that.

11

Precisely what I was thinking, and even if process 12345 was still running, it might have done pretty much anything before @TheJ started checking. Plus I'm not even sure how to check such things without getting into inotify / auditd / strace / dtrace type tooling (assuming Linux).

The wording of

can mean a few things. Perhaps you mean a bind mount? Something like:

services:
  es01-hot:
    volumes:
      - /data/es01-hot:/usr/share/elasticsearch/data
  es02-hot:
    volumes:
      - /data/es02-hot:/usr/share/elasticsearch/data
  es03-hot:
    volumes:
      - /data/es03-hot:/usr/share/elasticsearch/data
...

If so, all Elastic containers are running on the same (single) docker host?

@TheJ - what is your current status? If still unresolved, sharing the compose.yml and other config files is likely the most helpful thing you can do now.