Will filebeat lock files?

Elastic Stack Beats
1

Hi, I met a situation where it seemed that filebeat locked the application logs.

The application generates log file like appx.log, after some time or certain size is hit, it will rename the existing appx.log to appx_back.log, and start a new appx.log for logging. when it's time to rename the new appx.log again, it will just rename it and overwrite the existing appx_back.log.

With below filebeat settings, it is observed that application is not able to rename appx.log to appx_back.log, util appx.log reached a few GB, resulting in application crash. Does filebeat actually lock the log files it looks after? for this case, how to resolve it? thank you.

filebeat:
prospectors:
paths:
- //var/log/appx/*.log
input_type: log

  ignore_older: 10m
  document_type: appx
  tail_files: true

registry_file: "C:/ProgramData/filebeat/registry"

2

Filebeat keeps the file handler open until it reaches ignore_older (or close_older in the newer versions). But that should not prevent you from renaming the file, only from deleting it.

Which version of filebeat are you using? Which OS?

3

I'm using filebeat version 1.1.1 (386), on a window 2012 box.

> But that should not prevent you from renaming the file, only from deleting it.
if the appx_back.log already exists, the next rename from appx.log to appx_back.log involves an overwrite of the exsiting appx_back.log, not sure this is considered as delete?

4

I think that counts as a delete as the file will disappear afterwards.

5

But that doesn't mean filebeat will lock the file forever right?
the application checks whether appx.log needs to be renamed periodically, it can't be every time it checks, the file is locked by filebeat. Is there a way we can further troubleshoot this issue and solve it?

6

Filebeat should release the file handle after ignore_older? Can you update the version to 1.2.3? There is also close_older which can be set to a much lower value and ignore_older is not needed.

If you enable the debug output with -e -d "*" you should see when filebeat closes a file (or not).

7

Ok, I'll try it out

8

This topic was automatically closed after 21 days. New replies are no longer allowed.