Logstash filling daemon.log

Elastic Stack Logstash
1

Hi,

Since two days I run into a huge problem with my logstash.
It runs fine during the day, but at night it runs crazy like feeding a mogwai after midnight.
and I always get the same messages in my daemon.log:

Oct  5 06:25:14 mypc logstash[13329]: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
Oct  5 06:25:14 mypc logstash[13329]: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `run'
Oct  5 06:25:14 mypc logstash[13329]: 2018-10-05 06:25:14 +0200: Listen loop error: #<IOError: Too many open files>
Oct  5 06:25:14 mypc logstash[13329]: org/jruby/RubyIO.java:3705:in `select'

And when I check my server in the morning, I have a daemon.log of 12gb.
My current logstash version is 5.6.1.

2

How many open files does the Logstash process have (check with lsof)? What kind of files are they?

3

They are TCP-Connections and I checked it with lsof | grep "logstash.*TCP" | wc -l
I get 1268904 Files.

4

So it's leaking sockets. What inputs and outputs do you have? Does Logstash have thousands of open connections according to netstat?

5

Currently 1269060 open files and with netstat -an | wc -l I get a total of 16671.
For the inputs, I have some HTTPD logs and my output is just going sending to elasticsearch.

6

But what I have is a large amount of CLOSE_WAIT stati

7

And we use filebeat to send the logs to logstash.

8

I found our problem. My colleagues changed the certificate on our balancer in front of our elastics. That was the problem.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.