Filebeat paths logic


(Arthur Francis) #1

Hi,

Are there any other alternatives to using a wildcard (*) in filebeat when defining a log path. My application generates a numeric directory tree based on its activity lifecycle 34..35..36 etc

The amount of directories can get pretty high (over a thousand) and using a wildcard makes beats extremely slow to start as it can take several hours to scan all the paths. This makes it very impractical if I have config changes for example.


(Magnus Bäck) #2

Perhaps you can use a cron job or similar to periodically generate a new configuration file with more exact filename patterns?


(Arthur Francis) #3

Thanks @magnusbaeck I will give that a try.

Does the ignore_older setting also includes directories? Will this work in omitting the older directories which have already been processed or does it still need to do the preliminary scan?


(Magnus Bäck) #4

I'd expect ignore_older to only apply to files.


(Andrew Kroh) #5

Here is the matching that the filepath glob supports: https://golang.org/pkg/path/filepath/#Match

You can test your patterns here: https://play.golang.org/p/rJZII1vlJW


(Arthur Francis) #6

@magnusbaeck thank you, I managed to write a script that generates new config with relevant directories and reloads filebeat.

Hopefully the config reload feature will come in soon


(system) #7

This topic was automatically closed after 21 days. New replies are no longer allowed.