Recursive glob pattern depth

Elastic Stack Beats
1

We are trying to recursively capture logs from a file tree that gets quite deep. According to the documentation here: filestream input | Filebeat Reference [8.8] | Elastic
This states that: "If enabled it expands a single ** into a 8-level deep * pattern."

Is there any way to increase the depth? Using multiple ** does not work. e.g. /**/**/

2

Can you share an example of the path and what is your configuration?

From the documentation you just need to set it to /path/** and it will expand until 8 levels deep, if your path is more than 8 levels deep you will need to adjust and use the ** further in the path, something like /path/level1/level2/level3/**.

3

Hey. Thanks for the reply. Our current path configuration is:

        paths:
          - /var/jenkins_home/jobs/**
          - /var/jenkins_home/jobs/directory_1/jobs/**
        prospector.scanner.include_files: ['.*/log$']

We are capturing log files from all jobs running in our Jenkins instance.
Given how Jenkins creates it's directory structure we can capture up to three directories deep. Like this:

/var/jenkins_home/jobs/directory_1/jobs/directory_2/jobs/directory_3/jobs/job_name/builds/1/log

Adding directory_1 to our filebeat config is OK, but adding more than that makes the configuration too static. There is also no limitation on how many subfolders can be used within Jenkins.
So if we had another job with a path like this, we would not be able to capture it:

/var/jenkins_home/jobs/directory_1/jobs/directory_2/jobs/directory_3/jobs/directory_4/job_name/builds/1/log

Hope that helps clarify our use-case.

4

Not sure if there is something you can do besides add a lot of static prefixes.

Filebeat has this limitation on the recursive glob that will expand 8 levels only and I'm not sure you can add other globs in the path.

So you would need to add the static prefixes or depending on your needs use other log collector tool, If i'm not wrong vector from datadog does not have this glob expansion limitation and can also ship to Elasticsearch.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.