# Recursive glob pattern depth

**URL:** <https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 20, 2023, 11:26am UTC](https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471 "2023-06-20T11:26:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![unknotted-evacuee](https://avatars.discourse-cdn.com/v4/letter/u/cc9497/32.png) [@unknotted-evacuee](https://discuss.elastic.co/u/unknotted-evacuee)\
**Post date:** [June 20, 2023, 11:26am UTC](https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471/1 "2023-06-20T11:26:56Z")

</div>

We are trying to recursively capture logs from a file tree that gets quite deep. According to the documentation here: [filestream input | Filebeat Reference [8.8] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filestream-recursive-glob)  
This states that: "If enabled it expands a single `**` into a 8-level deep `*` pattern."

Is there any way to increase the depth? Using multiple \*\* does not work. e.g. `/ **/** /`

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 20, 2023, 12:30pm UTC](https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471/2 "2023-06-20T12:30:46Z")

</div>

Can you share an example of the path and what is your configuration?

From the documentation you just need to set it to `/path/ **` and it will expand until 8 levels deep, if your path is more than 8 levels deep you will need to adjust and use the `** ` further in the path, something like `/path/level1/level2/level3/**`.

---

<div class="post-metadata">

**Author:** ![unknotted-evacuee](https://avatars.discourse-cdn.com/v4/letter/u/cc9497/32.png) [@unknotted-evacuee](https://discuss.elastic.co/u/unknotted-evacuee)\
**Post date:** [June 20, 2023, 12:48pm UTC](https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471/3 "2023-06-20T12:48:50Z")

</div>

Hey. Thanks for the reply. Our current path configuration is:

```auto
        paths:
          - /var/jenkins_home/jobs/**
          - /var/jenkins_home/jobs/directory_1/jobs/**
        prospector.scanner.include_files: ['.*/log$']

```

We are capturing log files from all jobs running in our Jenkins instance.  
Given how Jenkins creates it's directory structure we can capture up to three directories deep. Like this:

```auto
/var/jenkins_home/jobs/directory_1/jobs/directory_2/jobs/directory_3/jobs/job_name/builds/1/log

```

Adding `directory_1` to our filebeat config is OK, but adding more than that makes the configuration too static. There is also no limitation on how many subfolders can be used within Jenkins.  
So if we had another job with a path like this, we would not be able to capture it:

```auto
/var/jenkins_home/jobs/directory_1/jobs/directory_2/jobs/directory_3/jobs/directory_4/job_name/builds/1/log

```

Hope that helps clarify our use-case.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 20, 2023, 1:04pm UTC](https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471/4 "2023-06-20T13:04:27Z")

</div>

Not sure if there is something you can do besides add a lot of static prefixes.

Filebeat has this limitation on the recursive glob that will expand 8 levels only and I'm not sure you can add other globs in the path.

So you would need to add the static prefixes or depending on your needs use other log collector tool, If i'm not wrong vector from datadog does not have this glob expansion limitation and can also ship to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2023, 3:05pm UTC](https://discuss.elastic.co/t/recursive-glob-pattern-depth/336471/5 "2023-07-18T15:05:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.