java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs

Elastic Stack Elasticsearch
1

Hi,

I have a problem with one of my elasticsearch node. For some reason node was shutdown due to some error. When I look into the log, I get the error java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs, see full error below.

{
  "@timestamp": "2026-09-08T09:38:55.683Z",
  "log.level": "ERROR",
  "message": "fatal exception while booting Elasticsearch",
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.server",
  "process.thread.name": "main",
  "log.logger": "org.elasticsearch.bootstrap.Elasticsearch",
  "elasticsearch.node.name": "es03-hot",
  "elasticsearch.cluster.name": "s1em",
  "error.type": "org.elasticsearch.ElasticsearchException",
  "error.message": "Failed to bind service",
  "error.stack_trace": "org.elasticsearch.ElasticsearchException: Failed to bind service\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:276)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.Node.<init>(Node.java:192)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)\nCaused by: org.apache.lucene.index.CorruptIndexException: Problem reading index. (resource=/usr/share/elasticsearch/data/_state/_pu2t.cfs)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:167)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:94)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:820)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:353)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:611)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:334)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:500)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)\n\t... 4 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:224)\n\tat java.base/java.nio.channels.FileChannel.open(FileChannel.java:309)\n\tat java.base/java.nio.channels.FileChannel.open(FileChannel.java:369)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:78)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:87)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)\n\t... 16 more\n"
}

When I look into /usr/share/elasticsearch/data/_state/ catalog, the file is actually missing.

How can I resolve that error?

Thanks in advance.

2

Hi @TheJ,

Can you share which version of Elasticsearch you are using? Have you recently upgraded? Do you have any background processes running that are deleting any files?

Let us know!

3

Thank you for quick response.

My elasticsearch cluster is in 8.13.0 version. I check for other processes on that server and there is only elasticsearch with docker so this isn't possible to delete that file by some process. Also, I'm 100% sure that file has not been deleted manually.

4

Can you share how you are starting your elasticsearch container?

5

Does this file exist on other nodes? [Please do not copy manually _pu2t.cfs]

6

Sure, I have docker compose configuration file and this cluster is deployed with docker stack deploy --compose-file compose.yml s1em command.

Also I have mounted volume for data on host machine (not as docker volume).

7

Nope, other nodes work without any problems and they are don't have that file

8

Please share your compose.yml file and other relevant configuration files.

9

Yeah, the specific file name is not that important here, as it would be a different something.cfs filename on other nodes in your cluster. More is that elasticsearch expects to find that file and does not find it. Is there anything in _state director on that node?

Yeah, the full error and sequence that led to node crash is likely more interesting. Including timestamps. "fatal exception while booting Elasticsearch" tells us it wont boot, I want to know why it crashed.

Sorry, experience tells me not to share your "isn't possible" confidence. It's possible, maybe unlikely, TBD.

10

100% this.

The only way to get Elasticsearch into a state where it needs to read this file is if it previously created this file and it definitely hasn't deleted it. The creation of this file included successful responses from fsync() indicating that this operation had become durable. The non-deletion of this file is because Elasticsearch would first have performed equally-durable writes indicating the file is no longer needed.

It is overwhelmingly likely that the explanation is that something on your system deleted this file out from underneath Elasticsearch. It may not be a process which is still running, so checking for other processes on the same machine is not conclusive.

One other possibility is that your storage subsystem reports success from writes and fsync() operations before those writes have become durable, which can become problematic if there were a power outage or other hardware failure at the wrong moment. But you didn't mention such a failure so it's probably not that.

11

Precisely what I was thinking, and even if process 12345 was still running, it might have done pretty much anything before @TheJ started checking. Plus I'm not even sure how to check such things without getting into inotify / auditd / strace / dtrace type tooling (assuming Linux).

The wording of

can mean a few things. Perhaps you mean a bind mount? Something like:

services:
  es01-hot:
    volumes:
      - /data/es01-hot:/usr/share/elasticsearch/data
  es02-hot:
    volumes:
      - /data/es02-hot:/usr/share/elasticsearch/data
  es03-hot:
    volumes:
      - /data/es03-hot:/usr/share/elasticsearch/data
...

If so, all Elastic containers are running on the same (single) docker host?

@TheJ - what is your current status? If still unresolved, sharing the compose.yml and other config files is likely the most helpful thing you can do now.

12

Hi,

sorry for delay. Here is my compose configuration

  es03-hot:
    image: elastic/elasticsearch:8.13.0
    hostname: es03-hot
    restart: always
    deploy:
      placement:
        constraints:
          - "node.hostname==node03"
      resources:
        limits:
          cpus: '17'
          memory: 64G
    ports:
      - published: 9331
        target: 9300
        protocol: tcp
        mode: host
    environment:
      - node.name=elasticsearch03-hot
      - cluster.name=elasticsearch
      - node.roles=data_content,data_hot
      - discovery.seed_hosts=${DISCOVER}
      - ELASTIC_PASSWORD=[somepassword]
      - network.host=0.0.0.0
      - transport.publish_host=${SERVER_3_IP}
      - transport.publish_port=9331
      - bootstrap.memory_lock=true
      - search.max_buckets=250000
      - xpack.monitoring.collection.interval=30s
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/elasticsearch03-hot/elasticsearch03-hot.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/elasticsearch03-hot/elasticsearch03-hot.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/elasticsearch03-hot/elasticsearch03-hot.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/elasticsearch03-hot/elasticsearch03-hot.key
    volumes: ['/ssd/data_hot:/usr/share/elasticsearch/data','certs:$CERTS_DIR']
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 131072
        hard: 131072
      nproc: 8192
      fsize: -1
13

sorry for delay, I still debugging but I don't find any solution.

Ps. Below i post compose configuration

14

Yes there is a lot of files like _r4lu.si or _r4ni_1.liv.

Here is full error

{"@timestamp":"2026-09-17T11:16:32.789Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch03-hot","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.ElasticsearchException","error.message":"Failed to bind service","error.stack_trace":"org.elasticsearch.ElasticsearchException: Failed to bind service\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:276)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.Node.<init>(Node.java:192)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)\nCaused by: org.apache.lucene.index.CorruptIndexException: Problem reading index. (resource=/usr/share/elasticsearch/data/_state/_pu2t.cfs)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:167)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:94)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:820)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:353)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:611)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:334)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:500)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)\n\t... 4 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/_state/_pu2t.cfs\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:224)\n\tat java.base/java.nio.channels.FileChannel.open(FileChannel.java:309)\n\tat java.base/java.nio.channels.FileChannel.open(FileChannel.java:369)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:78)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:87)\n\tat org.apache.lucene.core@9.10.0/org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)\n\t... 16 more\n"}

Let's say I'm 99% sure that file has not been deleted manually. Or bash history is lying to me

15

Checking current processes and bash history is unconvincing. You need to check all other possibilities too.

(note that by default preceding a command by a space in Bash means that it doesn't get saved to history)

16

@TheJ

Sorry, I’m on holiday so don’t have access to my usual tools, just peering at a mobile phone screen.

So maybe I missed it, but did you share the compose file ?

EDIT- see it now, but you shared only a single nodes section. What about the rest?

And I asked for the errors which led to the original crash, rather than those from subsequent failures to boot / start Elasticsearch.

Time has also moved on. You have other nodes, and a working cluster ? Is this now an ongoing lack of service issue, or just a curious little mystery you are trying to solve ?

The “looking at the shell history” test is nowhere near enough to rule out lots of other possibilities here. Honestly , I’d be checking all the layers between your effective virtual machine and the actual disk. And all the configuration. It’s quite easy to miss that there’s say a security scanner running somewhere with access to the filesystem. Ran periodically but not continuously. And loads of other possibilities too.