Filebeat tail_files in filestream input

Elastic Stack Beats
1

Hi All,

Starting from filebeat version 7.15 there is a recommendation to switch from log input type to filestream as log is deprecated.

My problem is that filestream apparently does not includes tail_files config option, which I need because I have large files and do not want to ingest old log lines. Only new ones. With filestream there is ignore_older option but I need to pull logs from Windows and as docs mention, sometimes Windows doesn't update file timestamp...

Any recommendations? Should I just stick with log input type?

2

filestream does provide a similar functionality, but it has been folded into ignore_inactive: filestream input | Filebeat Reference [7.17] | Elastic

Let me know if that solves your problem.

3

Thanks, @kvch although I'm not sure if this will help. As mentioned, I need to pull files from Windows:

The setting relies on the modification time of the file to determine if a file is ignored. If the modification time of the file is not updated when lines are written to a file (which can happen on Windows), the setting may cause Filebeat to ignore files even though content was added at a later time.

So correct me please If I misunderstood, but with this setting:

  1. ignore_inactive: since_first_start remembers time when Filebeat was first started.
  2. Sets offset in files that were never harvested at the end.
  3. Waits for timestamp of marked files to change - has to be newer than saved value of since_first_start.

So if Windows never updates timestamp of marked files, which as we know can happen, logs will never be pulled?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.