Filebeat doesn't update rotated log file's new content

[Haley]

Hi,

I need your help.
I have in trouble with dealing with filebeat.

Until now, I created new log files every day like
/home/haley/2017-01/01/session.log
/home/haley/2017-01/02/session.log
/home/haley/2017-01/03/session.log
...

and filebeat monitored it like this

paths:
- /home/haley///session.log
encoding: UTF-8
input_type: log
document_type: "XXXXX"
scan_frequency: 300s

because this way makes filebeat to monitor too much files and registry file bigger, I started to rotate log file. 1 time per 1 day like below. so

(batch such as)
cat /dev/null > /home/haley/session.log
cat /home/haley/2017-03/08/session.log > /home/haley/session.log

and changing path like

paths:
- /home/haley/session.log
encoding: UTF-8
input_type: log
document_type: "XXXXX"
scan_frequency: 300s

so basically, /home/haley/session.log 's inode doesn't change.
I thought filebeat will update new content in log file to KIBANA but it didn't work.

1. I found that if I modify /home/haley/session.log by vi command, then inode is changed then filebeat updates new content in log file to KIBANA right away.

2. I found that if I delete .filebeat (registry file) and restart filebeat, then filebeat writes registry file again and update whole /home/haley/session.log so entries are duplicated..

I just want filebeat to monitor only 1 log file(same inode) and
I just want filebeat to update new content in this log file to KIBANA.

Do you know how to solve this problem? or is it a bug?
my filebeat version is filebeat version 1.2.2 (amd64)

[ruflin]

First I strongly recommend you to update to Filebeat 5.x. There you can use the clean_* options to keep your registry file under control.

Filebeat is tracking inode and offset of a file. So if you modify content of a file and the inode and offset stay the same, for filebeat it is the same file. Otherwise filebeat would be required to keep the complete file in memory and compare it each time.

Modifying content instead of appending is not a typical log use case. Log truncation works in case the same file is reused and is smaller then before. But that can lead to potential data loss. I strongly recommend you to go back to your first use case where you monitor all log files instead of trying to push content form one file to an other.