I have an issue with a cluster setup. The setup has 3 nodes, all communicating via stunnel ( as per https://www.elastic.co/guide/en/cloud/current/ec-tunneling-ssl.html ).
However, the link is flaky from time to time, leading to a disconnect.
The link is down for 1-10 seconds at most, and syncing continues afterwards.
However, during this period it seems like the 'transactionlog' files are not closed properly.
Seeing as the disconnects are quite frequent this leads to a LOT of open files by elasticsearch ( all transactionlog files ) causing it to crash when it hits the ulimit.
The weird thing is, the cluster health stays at 100% all the time, not complaining about missing data or the likes.
Is there a way to have elasticsearch clean up these open files from time to time?
Thanks in advance!