We're running Logstash 7.16.2. We're using a TCP input that opens up too many sockets without closing them and almost every 24 hours Logstash hangs up.
[2022-01-05T05:10:59,474][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler java.io.IOException: Too many open files
We have even tried increasing the ulimit but that doesn't help, is this a bug in the tcp input plugin? We observed this problem with Logstash 7.15.2 as well.
Do you have a load balancer driving data into Logstash? If so, it could be due to a configuration problem in a load balancer. Is the Load balancer correctly closing connections? It may be worth reviewing a tcpdump to see what is happening.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.