# Logstash crashing with "Too many files open" **URL:** \ **Category:** Logstash\ **Created:** [December 6, 2015, 7:41pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479 "2015-12-06T19:41:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![vtst2412](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vtst2412/32/6228_2.png) [@vtst2412](https://discuss.elastic.co/u/vtst2412)\ **Post date:** [December 6, 2015, 7:41pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/1 "2015-12-06T19:41:27Z")
I have two logstash nodes running with 8 cores each. However, they consistently crash (every 1 or 2 days) with "Too many files open". I've only been able to find cases of this message on ES, has anyone encountered this on logstash? ``` -bash-4.2# ulimit -n 8192 ``` > > > I worry that this has to do with the performance of my grok filters. Any help or input on how to proceed to investigate ore remedy would be appreciated. ---
**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\ **Post date:** [December 6, 2015, 8:28pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/2 "2015-12-06T20:28:39Z")
try to track number of open FDs: ls /proc//fd/ | wc -l maybe you need more logstash resources (CPU, open files descriptors, memory, NIC bandwidth...) f.ex. from more nodes running your filtering. Try to track and verify that your logstash uses that many fds (but most do as the error shows :), so try to raise the ulimit for this process/user/system (see this link)or add more resources ie. more processes, if you got cpu and memory maybe more processes on same node listen on different input ports or add a MQ in front of your logstash nodes... ---
**Author:** ![vtst2412](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vtst2412/32/6228_2.png) [@vtst2412](https://discuss.elastic.co/u/vtst2412)\ **Post date:** [December 6, 2015, 8:34pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/3 "2015-12-06T20:34:50Z")
I have ran logstash with this in `/etc/sysconfig/logstash` ``` LS_OPEN_FILES=65535 ``` However, logstash eventually crashed (after 3-4 days) with the same "Too many files open" message in `logstash.err`. So it seems that increasing the open files limit only prolong the inevitable. ---
**Author:** ![Matthew\_Prinvale](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@Matthew\_Prinvale](https://discuss.elastic.co/u/Matthew_Prinvale)\ **Post date:** [December 10, 2015, 6:43pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/4 "2015-12-10T18:43:23Z")
I'm having the same issues here. Everything latest (Logstash 2.1.0, ES 2, etc). I noticed Logstash was crashing after a while even after changing my ulimit from 1024 to 64,000. I was thinking about changing it to unlimited but this look to be a bug so I'm glad I didn't. When I did an lsof I saw tens of thousands of these: ``` `redacted:9200 (ESTABLISHED) LogStash: 3311 3572 logstash 42u IPv6 5071865 0t0 TCP redacted:32880->redacted:9200 (ESTABLISHED) ``` it seems like Logstash isn't closing them. Is there a work-around until a fix is in place? ---
**Author:** ![vtst2412](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vtst2412/32/6228_2.png) [@vtst2412](https://discuss.elastic.co/u/vtst2412)\ **Post date:** [December 10, 2015, 6:45pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/5 "2015-12-10T18:45:32Z")
I fixed it by turning off sniffing (in elasticsearch output). logstash was keeping the tcp sockets open and consuming fd rapidly. ---
**Author:** ![Matthew\_Prinvale](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@Matthew\_Prinvale](https://discuss.elastic.co/u/Matthew_Prinvale)\ **Post date:** [December 10, 2015, 6:48pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/6 "2015-12-10T18:48:25Z")
nice! I just checked and mine is most certainly enabled (true). Was are the complications for changing this boolean? ---
**Author:** ![vtst2412](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vtst2412/32/6228_2.png) [@vtst2412](https://discuss.elastic.co/u/vtst2412)\ **Post date:** [December 10, 2015, 7:12pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/7 "2015-12-10T19:12:50Z")
It allows logstash to "sniff" the cluster and discover other ES nodes that it can potentially use to forward events to. By disabling it, you are telling logstash to only use the ES nodes specified in the `hosts` field. It's probably not all that useful unless your ES cluster is super active in term of horizontal scaling (i.e. you add and remove ES nodes a lot). ---
**Author:** ![Matthew\_Prinvale](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@Matthew\_Prinvale](https://discuss.elastic.co/u/Matthew_Prinvale)\ **Post date:** [December 10, 2015, 7:24pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/8 "2015-12-10T19:24:37Z")
Great! I'm pointing to a load balancer anyways! Glad I'm working this out in dev! haha. I made the change so _fingers crossed_ ---
**Author:** ![vtst2412](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vtst2412/32/6228_2.png) [@vtst2412](https://discuss.elastic.co/u/vtst2412)\ **Post date:** [December 10, 2015, 7:43pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/9 "2015-12-10T19:43:38Z")
No need to cross fingers. You can confirm as soon as logstash is restarted with sniffing disabled. `lsof | grep "logstash.*TCP" | wc -l` Run it a few seconds apart (about 5s would do). If the number is not growing rapidly, you are golden. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:18am UTC](https://discuss.elastic.co/t/logstash-crashing-with-too-many-files-open/36479/10 "2017-07-06T05:18:59Z")