How to deal with log archivation?

[budm (Dmitrii)]

In our project we scheduled archiving of log-files so they do not get too large. But the problem is filebeat harvests files periodically depending on backoff and max_backoff parameters.

But what if some log entry is written and the task to archive logs is ran in between the attempts to harvest log files? Then I miss the log entry which might be fatal depending on what this entry contained.

Is there a "true filebeat way" to cope with the situation?

As a possible solution I could set the backoff and max_backoff intervals small enough so this situation won't happen. But this would cause quite a lot of overhead due to too often harvesting.

[ruflin]

In most cases, before a log archivation happens, files are first rotated. This gives filebeat some time to finish reading the file. In case this is not enough to finish reading, in 5.0 I would recommend to set close_removed to false. This will keep the file open and will not remove it, until filebeat finished reading the file. This should "solve" your issue.

BUT: In case filebeat cannot catch up during rotation I think you will have lots of other issues down the road as it seems probably your output is too slow. That means you are producing log files faster then can be consumed, so earlier or later filebeat will get stuck because it can't send events fast enough and events pile up.

Is the problem you described and actual problem you have or a more theoretical question?