# Logstash filling daemon.log

**URL:** <https://discuss.elastic.co/t/logstash-filling-daemon-log/151154>\
**Category:** Logstash\
**Created:** [October 5, 2018, 5:41am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154 "2018-10-05T05:41:33Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\
**Post date:** [October 5, 2018, 5:41am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/1 "2018-10-05T05:41:34Z")

</div>

Hi,

Since two days I run into a huge problem with my logstash.  
It runs fine during the day, but at night it runs crazy like feeding a mogwai after midnight.  
and I always get the same messages in my daemon.log:

```auto
Oct 5 06:25:14 mypc logstash[13329]: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
Oct 5 06:25:14 mypc logstash[13329]: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `run'
Oct 5 06:25:14 mypc logstash[13329]: 2018-10-05 06:25:14 +0200: Listen loop error: #<IOError: Too many open files>
Oct 5 06:25:14 mypc logstash[13329]: org/jruby/RubyIO.java:3705:in `select'

```

And when I check my server in the morning, I have a daemon.log of 12gb.  
My current logstash version is 5.6.1.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 5, 2018, 1:34pm UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/2 "2018-10-05T13:34:29Z")

</div>

How many open files does the Logstash process have (check with lsof)? What kind of files are they?

---

<div class="post-metadata">

**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\
**Post date:** [October 6, 2018, 8:34am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/3 "2018-10-06T08:34:48Z")

</div>

They are TCP-Connections and I checked it with `lsof | grep "logstash.*TCP" | wc -l`  
I get 1268904 Files.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 8, 2018, 7:30pm UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/4 "2018-10-08T19:30:19Z")

</div>

So it's leaking sockets. What inputs and outputs do you have? Does Logstash have thousands of open connections according to netstat?

---

<div class="post-metadata">

**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\
**Post date:** [October 9, 2018, 5:34am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/5 "2018-10-09T05:34:27Z")

</div>

Currently 1269060 open files and with `netstat -an | wc -l` I get a total of 16671.  
For the inputs, I have some HTTPD logs and my output is just going sending to elasticsearch.

---

<div class="post-metadata">

**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\
**Post date:** [October 9, 2018, 6:25am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/6 "2018-10-09T06:25:26Z")

</div>

But what I have is a large amount of CLOSE\_WAIT stati

---

<div class="post-metadata">

**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\
**Post date:** [October 9, 2018, 6:54am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/7 "2018-10-09T06:54:45Z")

</div>

And we use filebeat to send the logs to logstash.

---

<div class="post-metadata">

**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\
**Post date:** [October 9, 2018, 9:21am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/8 "2018-10-09T09:21:56Z")

</div>

I found our problem. My colleagues changed the certificate on our balancer in front of our elastics. That was the problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 6, 2018, 9:32am UTC](https://discuss.elastic.co/t/logstash-filling-daemon-log/151154/9 "2018-11-06T09:32:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
