Hi Ruflin,
Greetings of the day...
I am Amit and currently doing a POC with Filebeat + ELK stack to explore it's features to make sure that it suits with our requirements if we may implements the same with our products.
I also has the same requirement like Stecino described above. I do understand that current version of Filebeat does not support this feature (fetch all files recursively in all sub-folders of a folder) but right now is there is any workaround available for the same? If yes then please let us know.
I will try to explain this in details as much as I may...
We have more than 100 web based applications (numbers are keep growing) which generates two types of logs which are 'IIS logs' and 'application logs'. These applications are hosted in more than one Environment like PROD, DEV, SIT and UAT. Predefined path for both Logs are on each ENV are:
E:/LogMonitor/[Environment]/Application/[APP_NAME_YYMMDD.txt
E:/LogMonitor/[Environment]/IIS/W3SVC[APP_Pool_ID].log //App_Pool_Id is dynamic and will be generated when new application will be hosted in IIS.
We ensures that I our system will never generate the duplicate files in both file locations mentioned above so now I want to define the prospectors in Filebeat yml file like below:
E:/LogMonitor//.txt
E:/LogMonitor//.log
Note: I tried this with single and two prospectors but it did not work and then when search the documentation then I came to know this feature is not supported yet.
I want to achieve this scenario because of two reasons:
- I don't want to go to my production support team every time to define a new prospector for a newly hosted application which consumes plenty of time in our cases due to taking approvals from multiple channels.
- There may be chance that I have to host applications in an new pre-production environment. E.g. a new OAT ENV. In this case need to add a prospector manually.
Please suggest.
BR//
Amit