Logstash 6.4 send duplicates when logrotate


(Chumu) #1

Hi,

I'm using Logstash 6.4 (OS: CentOS 6.8) to send logs from files to a message queue (a bit alike Kafka).

The files are in below structure, every 15 minutes, "access.log" in "second_dir" will be moved to sub-directory "archive_dir", and renamed to "access%Y%m%d%H%M.log".

|--top_dir
|----second_dir
| access.log
|------archive_dir
| access201812071215.log

my logstash configuration is as below:
input {
file {
path => "/top_dir/**/access*.log"
start_position => "beginning"
sincedb_path => "somewhere.sincedb"
sincedb_clean_after => 5
}
}
filter {}
output {}

As per the document detailed, the sincedb will track the state by inode, and the file will not be reread after rotation. But I got exact two duplicates, and the time interval they arrive my message queue is 15 minutes. It seems that it reread the renamed "accessXXXX.log" when it moved from "access.log".

Could there anyone please help on this, I've tried all my efforts (read the document again and again, google, and so on)..., many appreciate!


(Guy Boertje) #2

Please upgrade the file input to version 4.1.8. This is a known bug, now fixed.

bin/logstash-plugin install logstash-input-file --version 4.1.8


(Chumu) #3

Thanks for quick reply, but my server bases in a private network, can't access internet.

Is there any installable package? I can download it and put it the server.

BTW: how to check the file input version no?


(Guy Boertje) #4

bin/logstash-plugin list

Read https://www.elastic.co/guide/en/logstash/6.5/offline-plugins.html


(Chumu) #5

Thank you very much! It helps a lot!