Logstash and open files question

Elastic Stack Logstash
1

Hi all,
I have a question regarding how Logstash, when reading directly from files.
Here is my input pluging conf:

input {
    file {
        path => "/data/sbc/elkdev/*"
        start_position => "beginning"
        discover_interval => 300
        ignore_older => 1800
    }
}

As far I understand Logstash will ignore files whose modification date is 30 min older... Only files that are new or have been modified in the last 30 minutes will be taken into account.
In addition, will check the files every 5 minutes. Is my interpretation correct?

In my scenario I have two or three new files every 30 min (those files once copied are not further modified). So, if my interpretation is correct Logstash will have only 2 or 3 logs files opened at the same time.

But If I inspect the list of open files (using the lsof command) I see that Logstash maintains older files than the ones in the last 30 min.
What I'm doing/interpreting wrong?
Thank you!
Regards
Ana

2

The close_older option?

3

Thank you @Badger!
I put the close_older to 1800 and now I only have 3 files opened for the logstash pid

anna@elk01:/data/sbc/elkdev/scripts # lsof -p 20462 | grep FRAUD
java    20462 logstash   61r      REG              253,5    453678 27132653 /data/sbc/elkdev/FRAUD_SBC_INT_201803081630_000038535
java    20462 logstash   62r      REG              253,5    484880 27132654 /data/sbc/elkdev/FRAUD_SBC_INT_201803081645_000038536
java    20462 logstash   65r      REG              253,5    464944 27132646 /data/sbc/elkdev/FRAUD_SBC_INT_201803081700_000038537

But for each file open I have 55 threads... Is that the normal Logstash behaviour?
Using lsof -p LS_PID

anna@elk01:/data/sbc/elkdev/scripts # lsof -p 20462 | grep FRAUD | wc -l
3

Using lsof -K -p LS_PID

anna@elk01:/data/sbc/elkdev/scripts # lsof -K -p 20462 | grep FRAUD | wc -l
165

Thank you!
Anna

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.