Logstash open_file handle issue For Linux

[guyboertje]

If you have 5000 files discoverable files and you set max_open_files => 65535 then close_older is irrelevant for the purposes of managing the "sliding window" because all the files fit in the window (it does not slide) but it can be relevant to a lower value of file handles open at the OS level.

You might want to script periodic sampling of lsof on Logstash while setting close_older to "20s", "10s", "5s" and "2s" to see whether/when the files opened by Logstash goes below 5000.

[guyboertje]

Because the actual value you should use is dependent on the volume and frequency of appended content and the rate at which docs can be put into Elasticsearch (filter complexity and ES load) of your scenario, it is not really possible for me to say "use X".

[Rocky_RK]

Thanks @guyboertje, this helps.