Hi,
I'm using Logstash 6.4 (OS: CentOS 6.8) to send logs from files to a message queue (a bit alike Kafka).
The files are in below structure, every 15 minutes, "access.log" in "second_dir" will be moved to sub-directory "archive_dir", and renamed to "access%Y%m%d%H%M.log".
|--top_dir
|----second_dir
| access.log
|------archive_dir
| access201812071215.log
my logstash configuration is as below:
input {
file {
path => "/top_dir/**/access*.log"
start_position => "beginning"
sincedb_path => "somewhere.sincedb"
sincedb_clean_after => 5
}
}
filter {}
output {}
As per the document detailed, the sincedb will track the state by inode, and the file will not be reread after rotation. But I got exact two duplicates, and the time interval they arrive my message queue is 15 minutes. It seems that it reread the renamed "accessXXXX.log" when it moved from "access.log".
Could there anyone please help on this, I've tried all my efforts (read the document again and again, google, and so on)..., many appreciate!