Filebeat tail_files in filestream input

Hi All,

Starting from filebeat version 7.15 there is a recommendation to switch from log input type to filestream as log is deprecated.

My problem is that filestream apparently does not includes tail_files config option, which I need because I have large files and do not want to ingest old log lines. Only new ones. With filestream there is ignore_older option but I need to pull logs from Windows and as docs mention, sometimes Windows doesn't update file timestamp...

Any recommendations? Should I just stick with log input type?

filestream does provide a similar functionality, but it has been folded into ignore_inactive: filestream input | Filebeat Reference [7.17] | Elastic

Let me know if that solves your problem.

Thanks, @kvch although I'm not sure if this will help. As mentioned, I need to pull files from Windows:

The setting relies on the modification time of the file to determine if a file is ignored. If the modification time of the file is not updated when lines are written to a file (which can happen on Windows), the setting may cause Filebeat to ignore files even though content was added at a later time.

So correct me please If I misunderstood, but with this setting:

  1. ignore_inactive: since_first_start remembers time when Filebeat was first started.
  2. Sets offset in files that were never harvested at the end.
  3. Waits for timestamp of marked files to change - has to be newer than saved value of since_first_start.

So if Windows never updates timestamp of marked files, which as we know can happen, logs will never be pulled?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.