Logstash filling daemon.log


Since two days I run into a huge problem with my logstash.
It runs fine during the day, but at night it runs crazy like feeding a mogwai after midnight.
and I always get the same messages in my daemon.log:

Oct  5 06:25:14 mypc logstash[13329]: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
Oct  5 06:25:14 mypc logstash[13329]: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `run'
Oct  5 06:25:14 mypc logstash[13329]: 2018-10-05 06:25:14 +0200: Listen loop error: #<IOError: Too many open files>
Oct  5 06:25:14 mypc logstash[13329]: org/jruby/RubyIO.java:3705:in `select'

And when I check my server in the morning, I have a daemon.log of 12gb.
My current logstash version is 5.6.1.

How many open files does the Logstash process have (check with lsof)? What kind of files are they?

They are TCP-Connections and I checked it with lsof | grep "logstash.*TCP" | wc -l
I get 1268904 Files.

So it's leaking sockets. What inputs and outputs do you have? Does Logstash have thousands of open connections according to netstat?

Currently 1269060 open files and with netstat -an | wc -l I get a total of 16671.
For the inputs, I have some HTTPD logs and my output is just going sending to elasticsearch.

But what I have is a large amount of CLOSE_WAIT stati

And we use filebeat to send the logs to logstash.

I found our problem. My colleagues changed the certificate on our balancer in front of our elastics. That was the problem.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.