Logstash file input periodically skips over files

Elastic Stack Logstash
1

I am attempting to use the file input module to "watch" a directory for inbound files. Gzipped log files are shipped to this directory from various remote hosts, and the file input picks them up, decompresses or processes them "as is" and sends data to some upstream collector - graylog, elasticsearch, etc..

ubuntu 24.0.1
logstash 8.15.3

It is mostly working but every now and then it will skip a file or set of files. I am pretty sure my config is valid b/c it works MOST of the time.

input {
    file {
        path => "/var/lib/logstash/data/*gz"
        start_position => "beginning"
        mode => "read"
        file_completed_log_path => "/var/lib/logstash/consumed.log"
        file_completed_action => "log_and_delete"
    }
}

Anyone have ideas how best to troubleshoot this and/or know if i'm running into any known issues?

Thanks

2

That could be inode re-use. A low value for sincedb_clean_after might help.

3

Is that possible given i'm getting files predictably every 10m from various sources?

How "low" do you suggest?... i think the def. is 2w.

4

maybe do 15 mins?

5

Yes. If you have logstash delete files after reading them then the inodes are freed up and on some filesystems that will put them into a cache to be reused. This can make re-use quite common.

I would base the value of sincedb_clean_after on the maximum time you ever expect a file to stay in /var/lib/logstash/data/

6

i wonder if i would be better of NOT log_and_delete, and allow them to hang 24h period and run a find /var/lib/logstash/data -delete -mtime +1 sort of thing?

7

my original thinking is that:

  • servers send logs every 10m at varying times
  • logstash consumes them just as soon as it can
  • applies groks, filters, transformations, etc...
  • sends data to upstream graylog for persistence
  • next() log file...

So, i am counting on logstash to consume them as quick as possible, then log_and_delete those it was able to process.

That's where i'm at... so, i just set sincedb_clean_after to 1h. i'll see what that brings me. Interesting challenge inode reuse is...