# Limit the number of config files logstash reads from a directory

**URL:** <https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249>\
**Category:** Logstash\
**Created:** [November 2, 2017, 8:54pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249 "2017-11-02T20:54:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dulanjan\_Katuwawela](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Dulanjan\_Katuwawela](https://discuss.elastic.co/u/Dulanjan_Katuwawela)\
**Post date:** [November 2, 2017, 8:54pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249/1 "2017-11-02T20:54:16Z")

</div>

I'll try and outline the problem we're trying to solve. We have a process that automatically writes out multiple files (the content, layouts, filenames are all unique and different) into a specific directory. The files can vary in size too (small to a few gigs). Assume there are 100 files in this directory. We would like logstash to process X files at a single time. We can dynamically create config files for each file into a separate directory and pass the directory into logstash, however it would attempt to load them all at once. Is this controllable?

Is there a way to loadbalance multiple instances of logstash that can monitor this one directory?

Thanks in advance!!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 2, 2017, 9:01pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249/2 "2017-11-02T21:01:05Z")

</div>

There's no way to do this based on numbers, you could do it on file name matching though though, eg A\*, B\*, 0\*, 1\* etc

---

<div class="post-metadata">

**Author:** ![Dulanjan\_Katuwawela](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Dulanjan\_Katuwawela](https://discuss.elastic.co/u/Dulanjan_Katuwawela)\
**Post date:** [November 3, 2017, 1:11pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249/3 "2017-11-03T13:11:06Z")

</div>

Is there a way to queue and load balance logstash in any form when processing multiple files?

---

<div class="post-metadata">

**Author:** ![Dulanjan\_Katuwawela](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Dulanjan\_Katuwawela](https://discuss.elastic.co/u/Dulanjan_Katuwawela)\
**Post date:** [November 3, 2017, 2:08pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249/4 "2017-11-03T14:08:47Z")

</div>

I was reading up about Persistent Queues? Can this feature be leveraged in any way?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 3, 2017, 11:10pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249/5 "2017-11-03T23:10:36Z")

</div>

Persistent queues won't solve this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 1, 2017, 11:11pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249/6 "2017-12-01T23:11:03Z")

</div>

Closed on December 1, 2017, 11:11pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.