Issue with open files

Elastic Stack Elasticsearch
1

Hi All,

I have an issue with a cluster setup. The setup has 3 nodes, all communicating via stunnel ( as per https://www.elastic.co/guide/en/cloud/current/ec-tunneling-ssl.html ).
However, the link is flaky from time to time, leading to a disconnect.
The link is down for 1-10 seconds at most, and syncing continues afterwards.

However, during this period it seems like the 'transactionlog' files are not closed properly.
Seeing as the disconnects are quite frequent this leads to a LOT of open files by elasticsearch ( all transactionlog files ) causing it to crash when it hits the ulimit.

The weird thing is, the cluster health stays at 100% all the time, not complaining about missing data or the likes.

Is there a way to have elasticsearch clean up these open files from time to time?

Thanks in advance!

2

@WiseOwl Which Elasticsearch version are you using?

3

Hi Nhat,

All nodes are running:

  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
4

Hi @WiseOwl, I think you hit the issue described in https://github.com/elastic/elasticsearch/issues/49970. That bug was fixed in Elasticsearch 7.7.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.