Issue with open files

[WiseOwl]

Hi All,

I have an issue with a cluster setup. The setup has 3 nodes, all communicating via stunnel ( as per https://www.elastic.co/guide/en/cloud/current/ec-tunneling-ssl.html ).
However, the link is flaky from time to time, leading to a disconnect.
The link is down for 1-10 seconds at most, and syncing continues afterwards.

However, during this period it seems like the 'transactionlog' files are not closed properly.
Seeing as the disconnects are quite frequent this leads to a LOT of open files by elasticsearch ( all transactionlog files ) causing it to crash when it hits the ulimit.

The weird thing is, the cluster health stays at 100% all the time, not complaining about missing data or the likes.

Is there a way to have elasticsearch clean up these open files from time to time?

Thanks in advance!

[nhat (Nhat Nguyen)]

@WiseOwl Which Elasticsearch version are you using?

[WiseOwl]

Hi Nhat,

All nodes are running:

  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },

[nhat (Nhat Nguyen)]

Hi @WiseOwl, I think you hit the issue described in https://github.com/elastic/elasticsearch/issues/49970. That bug was fixed in Elasticsearch 7.7.