TCP Input "Too many files open" IOException

We're running Logstash 7.16.2. We're using a TCP input that opens up too many sockets without closing them and almost every 24 hours Logstash hangs up.

[2022-01-05T05:10:59,474][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler java.io.IOException: Too many open files

We have even tried increasing the ulimit but that doesn't help, is this a bug in the tcp input plugin? We observed this problem with Logstash 7.15.2 as well.

Do you have a load balancer driving data into Logstash? If so, it could be due to a configuration problem in a load balancer. Is the Load balancer correctly closing connections? It may be worth reviewing a tcpdump to see what is happening.

Are you using the tcp input as a server or a client?

Hey @Alex_Marquardt no there's no load balancer, there's a direct connection between the client (rsyslog) and Logstash.

Hey @Badger we're using it as a server

I believe that in server mode it is up to the client to close the channel. The server will leave it open in case the client wants to use it again.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.