Filebeat scan frequency

Elastic Stack Beats
1

Hi,
In filebeat configuration file, there is scan_frequency option. But I am confused about is scan frequency define the scanning new files in the path or scanning updates(new lines) in a file.

If the scan frequency checks the new files, what about the scanning frequency new lines in a file?

Thank you,
Gizem

2

Merhaba gizem,

the setting actually does multiple things.
First and foremost it is the setting that tells your beat when to check your log directory (lets say its /var/log/*.log) for new files to harvest, you are right about that.

The setting is also used to pick up closed files again.

If the scan frequency checks the new files, what about the scanning frequency new lines in a file?

Your filebeat will start a harvester for each log file you specified. This harvester will catch EVERY new line added to the log. When the harvester does not detect a new line for a specified timeframe (I think the default here is 5 minutes) it will close the file due to inactivity.
Another thing the scan_frequency option does is tell your harvester when to open the file again.

You can read more about it here:
General scan_frequency information:

Scan frequency and closed files

I hope this helps!

3

Merhaba @madduck,
It helps, thank you very much. As I understood, the log file remains open as long as a new line is added. And the new lines are harvested immeliately. Are the lines harvested one by one or as a group?

4

For filebeat every new line is its own log entry, or rather for syslog based logs. You can make use of the aggregate filter plugin to combine log lines based on certain criteria like for example the AUID

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.