Logstash - file input plugin with huge number of files

Elastic Stack Logstash
1

Hi,

I need to dump logs from one of our application to elastic search.
The application generates a unique log file for each request. Once the request is completed, this log file will be never updated again. We generate around 1000 log files per minute during peak hours.

Planning to use file input with close_older set to 300 seconds.

Any flaws or drawbacks with this approach? Most of the examples talk about using file input with single file or less number of files considering roll over.

Regards,
Chandan

2

I think you will run out of file descriptors quite fast.

3

If you set close_older and ignore_older to something sufficiently short you should be okay, I think.

4

Thanks for the reply guys. Will try with close_order and ignore_older set to short value. Will update once I have results.