Filebeat paths logic


Are there any other alternatives to using a wildcard (*) in filebeat when defining a log path. My application generates a numeric directory tree based on its activity lifecycle 34..35..36 etc

The amount of directories can get pretty high (over a thousand) and using a wildcard makes beats extremely slow to start as it can take several hours to scan all the paths. This makes it very impractical if I have config changes for example.

Perhaps you can use a cron job or similar to periodically generate a new configuration file with more exact filename patterns?

Thanks @magnusbaeck I will give that a try.

Does the ignore_older setting also includes directories? Will this work in omitting the older directories which have already been processed or does it still need to do the preliminary scan?

I'd expect ignore_older to only apply to files.

Here is the matching that the filepath glob supports:

You can test your patterns here:

@magnusbaeck thank you, I managed to write a script that generates new config with relevant directories and reloads filebeat.

Hopefully the config reload feature will come in soon

This topic was automatically closed after 21 days. New replies are no longer allowed.