Logstash file input plugin duplicate events after log file rollover

Elastic Stack Logstash
1

I have an application writing a log file, with log4j through RollingFileAppender.
Logstash listen to that file (control-engine.log) in order to retrieve and generate events:
When log file rollover is happening, the fileInputPlugin generate all the events again for the rolled file. (the rolled file is one file from c:/logfiles/*/control-engine-yyyy-MM-dd-HH-mm.log)

If I remove the "c:/logfiles//control-engine.log" from path, when rollover happens, it will just process again the rollover-ed file but will not process the "new" control-engine.log file.

here is the configuration:

 file {

   path => ["c:/logfiles/control-engine.log", "c:/logfiles/*/control-engine*.log"]	
     sincedb_path => "sincedb_path"
  start_position => "end"   
  mode => "tail"

}

and the message which I can see in the console when rollover happens:

[WARN ][filewatch.tailmode.processor] >>> Rotation In Progress - inode change detected and original content is not fully read, file is closed and path points to new content 
{"watched_file details"=>"<FileWatch::WatchedFile: @filename='control-engine-2019-11-13-18-20-1.log', @state='rotation_in_progress',
 @recent_states='[:watched, :watched]', @bytes_read='0', @bytes_unread='0', current_size='3630', last_stat_size='3630', file_open?='false', @initial=false, @sincedb_key='3797575424-308141-917504 0 0'>"}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.