"AlreadyClosedException" with "Too Many Open Files" - breaking indexing

astrodi · July 27, 2026, 8:35am

Hello there

I'm experiencing allocation issues on cluster with 22 data nodes (K8S cluster, each node holds 29-41 shards).

In total, there is 122M documents, 13TB of storage consumed and 456 pri. shards + 328 replica shards.

Version of Elasticsearch is 8.19.4

Symptoms:

Elasticsearch cluster falls into yellow state (UNASSIGNED replica shards) during indexing phase
number of UNASSIGNED replica shards increases over time
issue is repetitive (when restart problematic nodes -> cluster green, but next indexing causing same trouble)
issue visible after added more indices into cluster (more shards)

- [indices:data/write/bulk[s][r]] Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed

- Too many open files in system

- failed to create shard, failure org.elasticsearch.ElasticsearchTimeoutException: timed out while waiting to acquire shard lock for ...

Monitored:

Open file count on ES side: "open_file_descriptors": 2461, "max_file_descriptors": 1048576 (checked on all nodes, looks OK)
Swap: swap.used = 0
Heap usage = 60-70%, RAM usage = 75-99%
monitor hot_threads (stuck write operations, shard waiting indefinitely for closing, NOT CPU problem)

Steps I tried to resolve the issue:

POST /_cluster/reroute?retry_failed&metric=none is NOT solving the issue (ends up on 5x retry and back unassigned)
set node_concurrent_recoveries = 2 (was 10)
increase refresh_interval = 30s (was 10s)
disable slow_logs
removed unnecessary indices and decreased shard-count
looks, that issue doesn't happen (so frequently) on non-replica indices

None of steps I tried helped to prevent the issue.

The only short-term solution was restart ES data nodes - but it is not resolving root-cause of the problem.

I got advice that this might be resources issue on OS level, but don't know where to focus & identify which resource to increase.

Thank you in advance for advises, what to check

Dominik

DavidTurner · July 27, 2026, 10:01am

This error (with the in system suffix) is how glibc renders the ENFILE error, which means you've hit some kind of system-wide limit on open files rather than anything specific to a single process. Compare:

github.com/bminor/glibc

manual/errno.texi

04e750e75


      
          @deftypevr Macro int ENFILE
          @standards{POSIX.1, errno.h}
          @errno{ENFILE, 23, Too many open files in system}
          There are too many distinct file openings in the entire system.  Note
          that any number of linked channels count as just one file opening; see
          @ref{Linked Channels}.  This error never occurs on @gnuhurdsystems{}.
          @end deftypevr

with EMFILE that doesn't have the "in system" suffix:

github.com/bminor/glibc

manual/errno.texi

04e750e75


      
          @deftypevr Macro int EMFILE
          @standards{POSIX.1, errno.h}
          @errno{EMFILE, 24, Too many open files}
          The current process has too many files open and can't open any more.
          Duplicate descriptors do count toward this limit.
          
          In BSD and GNU, the number of open files is controlled by a resource
          limit that can usually be increased.  If you get this error, you might
          want to increase the @code{RLIMIT_NOFILE} limit or make it unlimited;
          @pxref{Limits on Resources}.
          @end deftypevr

This aligns with "open_file_descriptors": 2461 which is a totally reasonable number of files for the Elasticsearch process to hold open.

RainTown · July 27, 2026, 10:46am

Complicates things a little bit ...

As @DavidTurner said, that’s a system error. Do you have full access to the Kubernetes environment, or is that looked after by another team?

For example, can you inspect /proc/sys/fs/file-nr and /proc/sys/fs/file-max on the Kubernetes nodes? Can you run kubectl exec commands? Do you know which container runtime your Kubernetes environment is using?

Troubleshooting this would, IMO, be a Unix/Linux/Kubernetes administration job rather than an “Elasticsearch team” job — noting that, in a ton of environments, but far from all, those are exactly the same people.

astrodi · July 27, 2026, 10:55am

@RainTown - yes thanks for that, I'm already there - /proc/sys/fs/file-nr seems to be problem. Now I'm looking for the way, how to increase it via our Terra-form scripts & with DevOps specialist

RainTown · July 27, 2026, 11:45am

Glad that you seem on right track, a "DevOps Specialist” sounds like the right guy/gal. But minor rant, and nothing to do with you @astrodi, but ....
I remember when IT had about ten job titles. The more we fragment IT into hyper-specialised titles/roles, the more people we need to fix a problem that one competent sysadmin could understand and likely solve in about 10 minutes. Sadly.

astrodi · July 27, 2026, 4:52pm

I was able to change it myself (helm/values-dev.yaml and helm/values-prod.yaml)

This was the code required:

sysctl:
  - name: sysctl
    image: *shell_image
    imagePullPolicy: Always
    command:
      - /bin/bash
      - -ec
      - |
        sysctl -w vm.max_map_count=262144 && sysctl -w fs.file-max=2097152
        sysctl vm.max_map_count=262144 && sysctl fs.file-max=2097152

    securityContext:
      privileged: true
      runAsUser: 0
      runAsGroup: 0

So far tested in DEV, will continue with PROD cluster

kubectl exec -n ire-elasticsearch ire-elasticsearch-data-0 -- cat /proc/sys/fs/file-nr

RainTown · July 27, 2026, 5:17pm

Brilliant, well done. There's likely other ways, but I hope that sorts it out. Sorry for minor rant above.

btw, not sure what the second line is adding, looks redundant to me.

Topic		Replies	Views
"Too many open files in system" error from Elasticsearch Elasticsearch	1	11323	August 1, 2019
Stuck with too many open files issue Elasticsearch	2	4258	September 25, 2019
Elasticsearch - Too many open files Elasticsearch	3	15547	March 18, 2016
Too many open files even after increasing limit Elasticsearch	7	579	November 12, 2013
Elastic too many open files Elasticsearch	5	1190	September 14, 2018

"AlreadyClosedException" with "Too Many Open Files" - breaking indexing

Related topics