TCP Input "Too many files open" IOException

Elastic Stack Logstash
1

We're running Logstash 7.16.2. We're using a TCP input that opens up too many sockets without closing them and almost every 24 hours Logstash hangs up.

[2022-01-05T05:10:59,474][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler java.io.IOException: Too many open files

We have even tried increasing the ulimit but that doesn't help, is this a bug in the tcp input plugin? We observed this problem with Logstash 7.15.2 as well.

2

Do you have a load balancer driving data into Logstash? If so, it could be due to a configuration problem in a load balancer. Is the Load balancer correctly closing connections? It may be worth reviewing a tcpdump to see what is happening.

3

Are you using the tcp input as a server or a client?

4

Hey @Alex_Marquardt no there's no load balancer, there's a direct connection between the client (rsyslog) and Logstash.

5

Hey @Badger we're using it as a server

6

I believe that in server mode it is up to the client to close the channel. The server will leave it open in case the client wants to use it again.