Filebeat paths logic

(Arthur Francis) #1


Are there any other alternatives to using a wildcard (*) in filebeat when defining a log path. My application generates a numeric directory tree based on its activity lifecycle 34..35..36 etc

The amount of directories can get pretty high (over a thousand) and using a wildcard makes beats extremely slow to start as it can take several hours to scan all the paths. This makes it very impractical if I have config changes for example.

(Magnus Bäck) #2

Perhaps you can use a cron job or similar to periodically generate a new configuration file with more exact filename patterns?

(Arthur Francis) #3

Thanks @magnusbaeck I will give that a try.

Does the ignore_older setting also includes directories? Will this work in omitting the older directories which have already been processed or does it still need to do the preliminary scan?

(Magnus Bäck) #4

I'd expect ignore_older to only apply to files.

(Andrew Kroh) #5

Here is the matching that the filepath glob supports:

You can test your patterns here:

(Arthur Francis) #6

@magnusbaeck thank you, I managed to write a script that generates new config with relevant directories and reloads filebeat.

Hopefully the config reload feature will come in soon

(system) #7

This topic was automatically closed after 21 days. New replies are no longer allowed.