Filebeat missing events because of Detected rename of a previously harvested file

Elastic Stack Beats
1

Filebeat version : 1.2.3
OS : CentOS 7 (Docker)

I have two type of log in path /cudetect/, and they are rotated to /cudetect/old/ at the same time.

In /var/log/filebeat/filebeat :

2016-08-30T09:20:08Z INFO Old file with new name found: /cudetect/old/pkgversion.log-20160829.1472459341 is no /cudetect/old/pkgversion.log-20160830.1472548801
2016-08-30T09:20:08Z INFO Detected rename of a previously harvested file: /cudetect/old/pkgversion.log-20160829.1472459341 -> /cudetect/old/pkgversion.log-20160830.1472548801

2

Not sure I can follow 100%. Is filebeat detecting the wrong files to be renamed? Can you share your config file? Could it be that you hit an inode reuse issue? How toes the "rotation / renaming" happen? Can you check if you have the same issue with the most recent build of filebeat? https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

3

Tanks for your answer and sorry for my unclear description.

Is filebeat detecting the wrong files to be renamed?
--yes, indeed no file was renamed, files are truncated from /cudetect/ to /cudetect/old/ by logrotate+crontab.

Some of my Filebeat Config:
filebeat:
prospectors:
-
paths:
- /cudetect/old/pkgversion.log*
input_type: log
document_type: pkgversionlog
-
paths:
- /cudetect/old/upgradeexception.log-*
input_type: log
document_type: upgradeexceptionlog

Could it be that you hit an inode reuse issue?
--I'm not sure, I guess it is.

How toes the "rotation / renaming" happen?
--logrotate+crontab, source file in /cudetect/ and truncated file in /cudetect/old/

BTW, the wrong detecting of file-renaming occurs occasionally. I tried change the two kind of file to different path: pkgversion.log was truncated to /cudetect/pkg_old and upgradeexception.log was truncated to /cudetect/upd_old, after I modified filebeat.yml and restarted filebeat, this occured:
2016-09-02T08:00:09Z INFO Old file with new name found: /cudetect/old/upgradeexception.log-20160829.1472457541 is no /cudetect/pkg_rotate/pkgversion.log-20160902.1472803201
2016-09-02T08:00:09Z INFO Detected rename of a previously harvested file: **/cudetect/old/**upgradeexception.log-20160829.1472457541 -> /cudetect/pkg_rotate/pkgversion.log-20160902.1472803201

4

I'm not sure we use the same meaning for truncation. For me truncation is if we have an existing file with 5 lines and all the content of the file is removed, but the file stays the same and does not change the name. Above you write that upgradeexception.log was truncated to /cudetect/upd_old. Was it moved?

In the standard usecase logrotate does just rename the file. What is you crontab doing? Moving the file to an other place? Or are you copying the file?

Independent of the above, it seems that a new file is detected as an old file (which I assume was already removed). This sounds a lot like inode reuse. To mitigate such issues, in filebeat 5.0 there are the clean_* config options: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#clean-options As you have a docker container, this is even more like as normally in a docker container almost nothing happens on the files, so if one is removed and a new one is created, it is very often with the old inode.

To verify this assumption, you would have to check the inode of your old files and compare it with the new ones to see if they are the same.

5

This topic was automatically closed after 21 days. New replies are no longer allowed.