Need recommended FileBeat config for logs that replace file every hour

[HackerHurricane (Michael)]

Hello all,

I have a tool that we slurp up the files to Humio and FileBeat works perfectly if I delete the file before the task runs, but we would like not to delete the file so we don't get all the lines every hour, just the new lines.

We are using the config below and it is working for releasing the handle so our tool runs and creates the file, but does not seem to check in or update on the hour when the file is updated if I don't delete the file.

So any hints would be great. Latest version in Windows 64 FileBeat

Thanks,

MG

- paths:
    - C:/Program Files/LOG-MD/Report_AutoRuns.csv
    - C:/Program Files/LOG-MD/Report_AutoRuns_Users.csv
  encoding: utf-8
  close_eof: true
  scan_interval: 10m
  fields:
    "type": LMD-AutoRuns

[steffens (Steffen Siering)]

Filebeat is supposed to tail files. Todo so, it keeps the last known offsets in the registry file. If filebeat finds a file is being updated (new file size), it will reopen the file and continue reading from the last known position.

Filebeat actively scans for file updates. The interval is configured by scan_interval. The time a scan starts depends on the starting time of filebeat. Scans are not rounded to the hour/minute.

[HackerHurricane (Michael)]

Yeah, due to the nature of hourly and daily file updates, this is my config

encoding: utf-16le
close_eof: true
scan_interval: 5m

I was looking more for other option. We do not keep the file open, so FileBeat creates a handle and then an hour later when our tool runs it can't write tot he file... breaking the process.

The above config is how we got around it.

But wondering if there were other option combos we could try.

MG

[steffens (Steffen Siering)]

For tailing a log file one needs shared access to the file. Filebeat requires read access only. The tool blocking the write if filebeat still has the file handle open makes me assume that your tool is accessing the file in exclusive mode. See CreateFile developer docs, if FILE_SHARE_READ is not set when opening the file, the tool and/or filebeat might block each other from time to time.

If you can not modify file access mode in your tool, best workaround is setting scan_interval to a lower value (as you already did). The scan_interval will be determine potential latencies.