Hi,
I added a new data node to my cluster with a different configuration from the others.
I have 3 data nodes that use a single "path.data" and it is working fine. The new one was configured to use multiple data paths, as follows:
path.data:
- /devops/data1
- /devops/data2
The fact is in this data node the number of open files just increase all the time and now is in ~400000 file descriptors.
ls /proc/29144/fd/ | wc -l
394819
I'm getting this node disconnected from my master nodes repeatedly and I'm assuming that open file descriptors is the reason:
[2020-08-17T12:47:08,398][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [elk05-data] no known master node, scheduling a retry
[2020-08-17T12:47:08,398][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [elk05-data] no known master node, scheduling a retry
[2020-08-17T12:47:08,593][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [elk05-data] no known master node, scheduling a retry
[2020-08-17T12:47:24,396][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [elk05-data] no known master node, scheduling a retry
[2020-08-17T12:47:24,396][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [elk05-data] no known master node, scheduling a retry
Am I missing something here? Any configuration that I should have made in first place?
Any reason to this server behave so differently from the ones with single data path?