We are using elastic 7.4.1 packaged from docker.elastic.co.
The workload is pretty straightforward, just a few replicas spread across a few nodes. Mostly reads. Plenty of CPU/mem. We update the index every 10 minutes. One thing that isn't ideal is that we have some data replicas set on the master nodes. When we created the cluster, we didn't set the master nodes as master-only.
Upon further inspection, it looks like these open filedescriptors started appear after we scaled down our kubernetes cluster (those replicas I mentioned).
I tried to force flush the translog, and while that cleared those open FD's, I see them building up again. Also, using a normal flush didn't seem to do anything.
Still no idea what might be causing this or how to even fix this.