# Logstash - file input plugin with huge number of files

**URL:** <https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105>\
**Category:** Logstash\
**Created:** [March 22, 2016, 1:10pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105 "2016-03-22T13:10:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chandan\_PR](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@Chandan\_PR](https://discuss.elastic.co/u/Chandan_PR)\
**Post date:** [March 22, 2016, 1:10pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105/1 "2016-03-22T13:10:31Z")

</div>

Hi,

I need to dump logs from one of our application to elastic search.  
The application generates a unique log file for each request. Once the request is completed, this log file will be never updated again. We generate around 1000 log files per minute during peak hours.

Planning to use file input with close\_older set to 300 seconds.

Any flaws or drawbacks with this approach? Most of the examples talk about using file input with single file or less number of files considering roll over.

Regards,  
Chandan

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [March 22, 2016, 3:06pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105/2 "2016-03-22T15:06:36Z")

</div>

I think you will run out of file descriptors quite fast.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 22, 2016, 6:08pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105/3 "2016-03-22T18:08:37Z")

</div>

If you set `close_older` and `ignore_older` to something sufficiently short you should be okay, I think.

---

<div class="post-metadata">

**Author:** ![Chandan\_PR](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@Chandan\_PR](https://discuss.elastic.co/u/Chandan_PR)\
**Post date:** [March 23, 2016, 1:34pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105/4 "2016-03-23T13:34:48Z")

</div>

Thanks for the reply guys. Will try with close\_order and ignore\_older set to short value. Will update once I have results.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:05am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105/5 "2017-07-06T05:05:42Z")

</div>