Logstash 6.3.2 - File input plugin 4.1.5 stops writing to .sincedb after some time

[ld_pvl]

Hi Team,

I have LS 6.3.2 installed with file input plugin 4.1.5 (manually upgraded from the 4.0.x shipped inside LS 6.3.2).

For context, the scenario I am dealing with is: the log files that are being monitored by this particular file input plugin have quite small size (max 50KB).

However, the application that writes these log files has very aggressive rolling strategy that makes the files appear and disappear in a very short amount of time (sometimes within a second or two) because they roll over each other.

So what happens over the long run is there are not a lot of actual physical files on the file system at a given point in time but there would have been numerous files that have been created and deleted in close successions throughout this period.

So I am try to track these files that appear and disappear very quickly using the config below which works on the best effort basis (i.e. I accept the fact that some logs might not be monitored in time if some files roll too quickly).

Now the problem is, LS sends the events from this file input fine but after some arbitrary amount of time it just stops sending the events even though events from other inputs in the same LS instance (such as heartbeat and other file inputs) are still being sent normally. Upon checking, I noticed that the problematic file input stops writing to the .sincedb file at the exact time that events stop being sent which lead me to think that there is a direct correlation here.

I tried playing with the sincedb_write_interval and noticed that the file input seems to be able to work for longer period if I decrease the sincedb_write_interval. However, I still do not understand the root cause as to why the file input would just stop writing to the .sincedb file because the debug logs do not show much either, I only see this single line repeating constantly without anything else:

[2018-08-28T11:46:10,115][DEBUG][filewatch.tailmode.handlers.grow] read_to_eof: get chunk

In the .sincedb file, I also see for some last lines in the last column (the file path column) there are no values.

I noticed that there is now a new version of LS which is 6.4 that just came out which ships now with 4.1.5 file input plugin but I do not think (though I'm not entirely certain) that it is a problem with LS version.

This is my config:

file {
  sincedb_path           => "/some/path/.sincedb_foo"
  sincedb_clean_after    => 1
  sincedb_write_interval => 1
  max_open_files         => 5000
  close_older            => 0.001
  ignore_older           => "1 d"
  path                   => "/path/logs/*"
  type                   => "my_type"
  start_position         => "beginning"
  discover_interval      => 1
  stat_interval          => 5

  codec => multiline {
    patterns_dir         => ["/some/dir"]
    pattern              => ".*"
    what                 => "previous"
    max_lines            => 1000
  }
}

Thanks for you help as usual.