Filebeat paths logic

Elastic Stack Beats
1

Hi,

Are there any other alternatives to using a wildcard (*) in filebeat when defining a log path. My application generates a numeric directory tree based on its activity lifecycle 34..35..36 etc

The amount of directories can get pretty high (over a thousand) and using a wildcard makes beats extremely slow to start as it can take several hours to scan all the paths. This makes it very impractical if I have config changes for example.

2

Perhaps you can use a cron job or similar to periodically generate a new configuration file with more exact filename patterns?

3

Thanks @magnusbaeck I will give that a try.

Does the ignore_older setting also includes directories? Will this work in omitting the older directories which have already been processed or does it still need to do the preliminary scan?

4

I'd expect ignore_older to only apply to files.

5

Here is the matching that the filepath glob supports: https://golang.org/pkg/path/filepath/#Match

You can test your patterns here: https://play.golang.org/p/rJZII1vlJW

6

@magnusbaeck thank you, I managed to write a script that generates new config with relevant directories and reloads filebeat.

Hopefully the config reload feature will come in soon

7

This topic was automatically closed after 21 days. New replies are no longer allowed.