Sincedb_write_interval not honoured during quiet period

[guido4f]

There is an issue with the updating of sincedb, if the files have received and completely order partially processed within a single sincedb_write_interval; and no subsequent records are processed the changes and read state is not reflected in sincedb as there are no subsequent events to trigger the check of the write interval

now = Time.now.to_i
delta = now - @sincedb_last_write
if delta >= @opts[:sincedb_write_interval]

This is demonstrable using the following pipeline and copying complete logs files across to the c:\work\logfiles\ directory.
After the content has been flushed and the files closed, if you look at the sincedb file there will be missing or incomplete records and waiting for them to appear does not help – as there are no events that trigger a sincedb update.

input {
  file {
    close_older => 20
     path => [
             "c:/work/logfiles/**/*.log*"
             ]
    start_position => "beginning"
    codec => multiline {
              pattern => "^(?!((?:\d{1,4})[\.\/-]\d{1,2}[\.\/-](?:\d{2}){1,2}[\sT]\d{1,2}:[0-5][0-9](:([0-5][0-9]|60))?))"
              what => "previous"
              max_lines => 100000
              charset => "UTF-8"
            }
  }
}
output {
  stdout { codec => rubydebug  }
}

This means that a non-graceful shutdown will result in the log being reread and reprocessed for further back than the sincedb_write_interval; which is not being honoured.

Could we get a sincedb write when the files are closed or the buffer flushed.