Logstash behavior on file name change

Elastic Stack Logstash
1

Hi, i configured logstasth to monitor a file with below config
file {
path => "/var/log/app1/xxxx.log"
type => "app1log"
}

when the file reaches certain size, it'll be renamed to xxxx-01.log by the application, and a new xxxx.log is created. I have two questions:

  1. obviously xxxx-01.log is not monitored, if logstash hasn't finished sending all contents before the file is renamed, what will happen to remaining loggings?
  2. if i change the path configuration to "/var/log/app1/xxxx*.log", the renamed file xxxx-01.log is also under monitor. Will logstash send all contents in xxxx-01.log regardless of how much it has sent before file rename? or it will only send the remaining, regardless of name change?
2

I looked into this the other day and started to document the file rotation behavior in https://github.com/logstash-plugins/logstash-input-file/pull/61. Based on the understanding I believe I gained there:

  1. If Logstash discovers the renaming of the file before all the data has been sent, you won't get all data.
  2. Logstash loses the sincedb entry for a file when it's renamed (i.e. it doesn't understand renames at all; they're treated as a deletion and an addition of a file). With start_position set to "end" (the default) the file will be tailed from the end as soon as Logstash discovers the file after the rename, so if the file is renamed and is written to for a non-trivial amount of time you will get that data. However, because of the preceding bullet, if Logstash isn't up to speed with the file prior to the rename you may still lose data. With start_position set to "beginning" Logstash will process the renamed file from the start.
1 Like
3

Thank you for the detailed explanation!
Is there a way to work around this so that I don't get any data loss?