# Filebeat: recursively fetch all files in all subdirectories of a directory **URL:** \ **Category:** Beats\ **Created:** [April 19, 2016, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834 "2016-04-19T20:29:32Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [April 19, 2016, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/1 "2016-04-19T20:29:32Z")
I thought i read somewhere that filebeat 5.0.0 alpha has addressed this. But now after implementation it says it doesn't ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [April 20, 2016, 5:46pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/2 "2016-04-20T17:46:21Z")
Since I wanted to go one directory down appending /_/_.log did the trick. But it would be nice for future filebeat releases to have recursive support. Splunk forwarder for example handles this well 🙂 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 21, 2016, 11:24am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/3 "2016-04-21T11:24:31Z")
I remember there were discussion for this feature but I don't there is somewhere state that this is part of 5.0. We currently rely on the `glob` implementation from golang which has the above limitation. Going with a recursive approach can have the affect that a very large amount of files are crawled by one prospector, splitting it up to multiple prospectors give us the ability to somehow bring this under control (no implementation either here yet). Could you share some more details on the structure of your log files and why you would need the recursive file fetching? If adding a feature, it is always good to understand the use case. ---
**Author:** ![ABEE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abee/32/9525_2.png) [@ABEE](https://discuss.elastic.co/u/ABEE)\ **Post date:** [May 3, 2016, 3:53am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/4 "2016-05-03T03:53:22Z")
Hi Ruflin, Greetings of the day... I am Amit and currently doing a POC with Filebeat + ELK stack to explore it's features to make sure that it suits with our requirements if we may implements the same with our products. I also has the same requirement like Stecino described above. I do understand that current version of Filebeat does not support this feature (fetch all files recursively in all sub-folders of a folder) but right now is there is any workaround available for the same? If yes then please let us know. I will try to explain this in details as much as I may... We have more than 100 web based applications (numbers are keep growing) which generates two types of logs which are 'IIS logs' and 'application logs'. These applications are hosted in more than one Environment like PROD, DEV, SIT and UAT. Predefined path for both Logs are on each ENV are: E:/LogMonitor/[Environment]/Application/[APP\_NAME\_YYMMDD.txt E:/LogMonitor/[Environment]/IIS/W3SVC[APP\_Pool\_ID].log //App\_Pool\_Id is dynamic and will be generated when new application will be hosted in IIS. We ensures that I our system will never generate the duplicate files in both file locations mentioned above so now I want to define the prospectors in Filebeat yml file like below: E:/LogMonitor/_/_.txt E:/LogMonitor/_/_.log Note: I tried this with single and two prospectors but it did not work and then when search the documentation then I came to know this feature is not supported yet. I want to achieve this scenario because of two reasons: 1. I don't want to go to my production support team every time to define a new prospector for a newly hosted application which consumes plenty of time in our cases due to taking approvals from multiple channels. 2. There may be chance that I have to host applications in an new pre-production environment. E.g. a new OAT ENV. In this case need to add a prospector manually. Please suggest. BR// Amit ---
**Author:** ![monica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monica/32/3696_2.png) [@monica](https://discuss.elastic.co/u/monica)\ **Post date:** [May 3, 2016, 12:42pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/5 "2016-05-03T12:42:01Z")
If you know all the available environments [Environement], a solution that I can think of is to create a prospector for each environment. Please correct me if I didn't understood correctly. 1rst prospector: - E:/LogMonitor/PROD/Application/\*.txt - E:/LogMonitor/PROD/IIS/\*.log 2nd prospector: - E:/LogMonitor/DEV/Application/\*.txt - E:/LogMonitor/DEV/IIS/\*.log 3rd prospector: - E:/LogMonitor/SIT/Application/\*.txt - E:/LogMonitor/SIT/IIS/\*.log 4th prospector: - E:/LogMonitor/UAT/Application/\*.txt - E:/LogMonitor/UAT/IIS/\*.log ---
**Author:** ![ABEE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abee/32/9525_2.png) [@ABEE](https://discuss.elastic.co/u/ABEE)\ **Post date:** [May 3, 2016, 1:51pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/6 "2016-05-03T13:51:48Z")
Hi Monica, Thanks for your inputs but this solution does not suits with my requirements. Actually each IIS folder has more than 100 sub folders and number is keep increasing because every time we host a new application it creates a new folder under IIS folder. It is true for all Env. This is why I am looking for a workaround that can read all log files from all sub-folders with single prospector. In my case it should be like below: D:/LogFiles/Dev/IIS/_/_.log Or D:/LogFiles/_/_.log. (Preferred option) Many Thanks. Amit. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 3, 2016, 3:07pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/7 "2016-05-03T15:07:37Z")
I haven't tried it, but I actually thought the following should work for you: ```auto E:/LogMonitor/*/Application/*.txt E:/LogMonitor/*/IIS/W3SVC*.log ``` What is not supported, that you can defined `E:/LogMonitor/**` and it will crawl all sub directories. ---
**Author:** ![ABEE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abee/32/9525_2.png) [@ABEE](https://discuss.elastic.co/u/ABEE)\ **Post date:** [May 4, 2016, 9:54am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/8 "2016-05-04T09:54:58Z")
Many thanks Ruflin. Wildcard solution work for me. I was not aware about this awesome workaround. Thanks a ton again. ---
**Author:** ![sruthib](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@sruthib](https://discuss.elastic.co/u/sruthib)\ **Post date:** [August 12, 2016, 7:04am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/9 "2016-08-12T07:04:14Z")
Hi all, I am using Filebeat to read logs from my PC and send it out to logstash. I have folder and sub folders in a random order. There is no specific order of sub directories which contain the logs. So can you suggest a way to use Filebeat to handle such situation. Folder1\Folder2\log1.txt Folder1\Folder2\Folder3\Folder4\log1.txt So i cannot explicitly mention the pattern of sub folders. Not to forget all the folders here also have logs that have to be read with the sub folders. Filebeat fails to do that using \*\* Any help is appreciated. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [August 12, 2016, 8:22am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/10 "2016-08-12T08:22:13Z")
This is currently not supported. Have a look at [https://github.com/elastic/beats/issues/2084](https://github.com/elastic/beats/issues/2084) ---
**Author:** ![sruthib](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@sruthib](https://discuss.elastic.co/u/sruthib)\ **Post date:** [August 12, 2016, 8:26am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/11 "2016-08-12T08:26:53Z")
Any workaround to get this done?? Or is there any other option or tool? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [August 12, 2016, 8:28am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/12 "2016-08-12T08:28:18Z")
Perhaps you can tackle this from the other way. Why does your application create log files in random folder structures? ---
**Author:** ![sruthib](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@sruthib](https://discuss.elastic.co/u/sruthib)\ **Post date:** [August 12, 2016, 8:33am UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/13 "2016-08-12T08:33:02Z")
There are different ways to pull the logs from the application. So it creates different file structures. We cannot handle it from the file structure side. Any other option that can help us? Is that even going to be a considered requirement for filebeat in the future release? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:50pm UTC](https://discuss.elastic.co/t/filebeat-recursively-fetch-all-files-in-all-subdirectories-of-a-directory/47834/14 "2017-07-05T21:50:46Z")