# Shipping log from an unmanagable folder structure using filebeat

**URL:** <https://discuss.elastic.co/t/shipping-log-from-an-unmanagable-folder-structure-using-filebeat/289014>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 11, 2021, 6:46pm UTC](https://discuss.elastic.co/t/shipping-log-from-an-unmanagable-folder-structure-using-filebeat/289014 "2021-11-11T18:46:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![luka.klaric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luka.klaric/32/71073_2.png) [@luka.klaric](https://discuss.elastic.co/u/luka.klaric)\
**Post date:** [November 11, 2021, 6:46pm UTC](https://discuss.elastic.co/t/shipping-log-from-an-unmanagable-folder-structure-using-filebeat/289014/1 "2021-11-11T18:46:09Z")

</div>

I am trying to ship log files to logstash using filebeat. The folder structure of the input path is currently very branchy and very big.  
We have a following structure:  
`<LOG_PATH>/<STREAM>/<WORKFLOW>/<TASK>/<EXECUTION_DATE>/<TRY_NUMBER>.log`

The logs are actually not that big, but the application produces every day an average of 300.000 new log files, depending which workflows are running and which tasks. Every newly created log creates a new execution\_date folder containing the log file, and at the end we are having an unmanageable structure. A 'find' through the folder can take a week or more, and a 'ls' in some workflow folder can take for hours. I even wrote a python script to count the files and task folders using glob, but the execution was terminated after some time.

As I could find out, filebeat is using filepath/glob, so my question is: how does filebeat "walk" through the input? can it be comparable to pythons glob, so that it just terminates after some time without any error log? can I assume that filebeat just can't handle this amount of files, under a shared volume?

I created once a related question [link](https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 9, 2021, 8:47pm UTC](https://discuss.elastic.co/t/shipping-log-from-an-unmanagable-folder-structure-using-filebeat/289014/2 "2021-12-09T20:47:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.