Logstash 6.4 send duplicates when logrotate


I'm using Logstash 6.4 (OS: CentOS 6.8) to send logs from files to a message queue (a bit alike Kafka).

The files are in below structure, every 15 minutes, "access.log" in "second_dir" will be moved to sub-directory "archive_dir", and renamed to "access%Y%m%d%H%M.log".

| access.log
| access201812071215.log

my logstash configuration is as below:
input {
file {
path => "/top_dir/**/access*.log"
start_position => "beginning"
sincedb_path => "somewhere.sincedb"
sincedb_clean_after => 5
filter {}
output {}

As per the document detailed, the sincedb will track the state by inode, and the file will not be reread after rotation. But I got exact two duplicates, and the time interval they arrive my message queue is 15 minutes. It seems that it reread the renamed "accessXXXX.log" when it moved from "access.log".

Could there anyone please help on this, I've tried all my efforts (read the document again and again, google, and so on)..., many appreciate!

Please upgrade the file input to version 4.1.8. This is a known bug, now fixed.

bin/logstash-plugin install logstash-input-file --version 4.1.8

Thanks for quick reply, but my server bases in a private network, can't access internet.

Is there any installable package? I can download it and put it the server.

BTW: how to check the file input version no?

bin/logstash-plugin list

Read https://www.elastic.co/guide/en/logstash/6.5/offline-plugins.html

Thank you very much! It helps a lot!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.