I am trying to rotate files nightly, where file is being renamed from name.log to name-{yyyy-mm-dd}.log and creating a new name.log file. In logstash input config, I am setting path => name*.log
In the documentation about file rotation here, it is quoted that
File rotation is detected and handled by this input, regardless of whether the file is rotated via a rename or a copy operation. ............... Note that the rotated filename will be treated as a new file so if start_position is set to beginning the rotated file will be reprocessed.
I don't see why the old file will be reprocessed again just by renaming and if its start_position is set to beginning. The inode of a file will remain same even if the file is renamed (I am not saving). And once the file offset is set in sincedb, 'start_position' parameter should not have any effect as quoted here
This option only modifies "first contact" situations where a file is new and not seen before, i.e. files that don’t have a current position recorded in a sincedb file read by Logstash. If a file has already been seen before, this option has no effect and the position recorded in the sincedb file will be used.
I am facing an issue with file rotation. I am sending logstash ouput to Rabbit MQ and that queue is going down over the weekend. So, in order to read the files written over the weekend, I am using start_position=>beginning, because files written over the weekend are new files and hence they should be read from the beginning.
But, with path => name*.log, start_position=>beginning and ignore_older=>86400, all the logs are getting duplicated. I see it is behaving as mentioned in 1st quote about file rotation (reading the rotated file again), but inode number of old file before and after rename is not changing in sincedb, so start_position=>beginning should not have any effect right?