Close_removed rotate not wroking properly issue

Elastic Stack Beats
1

Hi:
I encountered the issue when using filebeat 7.4.1 in kubernetes.
I set the rotate number of docker logs with 5, and enable close_removed, disable close_moved.
In this way, i want filebeat keep trace of the container logs until it has already rotated for 5 times. if the collection does not finish after a log file has rotate for 5 time, then the drop of logs is acceptable.
But the result is that filebeat do not release the file description until it reach the end of the log file. It continue to collect the log data.
if any app produce logs too fast to collect in time, this could fill the disk up.

so I make 2 test:

  1. copy a file to directory where filebeat collect data. then remove the file before filebeat collect all data of the file.
    In the test, filebeat stoped keeping trace of the file, and release the file description.

  2. copy a file to directory where filebeat collect data. when removing the file before filebeat finish collecting all data of the file, i create a empty file with the same file name in the directory.
    In the test, filebeat continue keep trace of the deleted file, and do not release the file description.

My question is :

  1. is this a bug of close_removed option
  2. if this is not a bug, is there any way i can make the droping of file which has rotated for certain time come true.

Regards,
theseaofstarts

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.