# Need recommended FileBeat config for logs that replace file every hour

**URL:** <https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 23, 2018, 2:34pm UTC](https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755 "2018-08-23T14:34:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![HackerHurricane](https://avatars.discourse-cdn.com/v4/letter/h/4da419/32.png) [@HackerHurricane](https://discuss.elastic.co/u/HackerHurricane)\
**Post date:** [August 23, 2018, 2:34pm UTC](https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755/1 "2018-08-23T14:34:02Z")

</div>

Hello all,

I have a tool that we slurp up the files to Humio and FileBeat works perfectly if I delete the file before the task runs, but we would like not to delete the file so we don't get all the lines every hour, just the new lines.

We are using the config below and it is working for releasing the handle so our tool runs and creates the file, but does not seem to check in or update on the hour when the file is updated if I don't delete the file.

So any hints would be great. Latest version in Windows 64 FileBeat

Thanks,

MG

```auto
- paths:
    - C:/Program Files/LOG-MD/Report_AutoRuns.csv
    - C:/Program Files/LOG-MD/Report_AutoRuns_Users.csv
  encoding: utf-8
  close_eof: true
  scan_interval: 10m
  fields:
    "type": LMD-AutoRuns

```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 27, 2018, 1:47pm UTC](https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755/2 "2018-08-27T13:47:25Z")

</div>

Filebeat is supposed to tail files. Todo so, it keeps the last known offsets in the registry file. If filebeat finds a file is being updated (new file size), it will reopen the file and continue reading from the last known position.

Filebeat actively scans for file updates. The interval is configured by `scan_interval`. The time a scan starts depends on the starting time of filebeat. Scans are not rounded to the hour/minute.

---

<div class="post-metadata">

**Author:** ![HackerHurricane](https://avatars.discourse-cdn.com/v4/letter/h/4da419/32.png) [@HackerHurricane](https://discuss.elastic.co/u/HackerHurricane)\
**Post date:** [August 27, 2018, 2:39pm UTC](https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755/3 "2018-08-27T14:39:31Z")

</div>

Yeah, due to the nature of hourly and daily file updates, this is my config

encoding: utf-16le  
close\_eof: true  
scan\_interval: 5m

I was looking more for other option. We do not keep the file open, so FileBeat creates a handle and then an hour later when our tool runs it can't write tot he file... breaking the process.

The above config is how we got around it.

But wondering if there were other option combos we could try.

MG

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 27, 2018, 8:49pm UTC](https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755/4 "2018-08-27T20:49:48Z")

</div>

For tailing a log file one needs shared access to the file. Filebeat requires read access only. The tool blocking the write if filebeat still has the file handle open makes me assume that your tool is accessing the file in exclusive mode. See [CreateFile developer docs](https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilea), if FILE\_SHARE\_READ is not set when opening the file, the tool and/or filebeat might block each other from time to time.

If you can not modify file access mode in your tool, best workaround is setting `scan_interval` to a lower value (as you already did). The `scan_interval` will be determine potential latencies.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 24, 2018, 9:03pm UTC](https://discuss.elastic.co/t/need-recommended-filebeat-config-for-logs-that-replace-file-every-hour/145755/5 "2018-09-24T21:03:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.