# Filebeat doesn't harvest folder structure

**URL:** <https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 11, 2021, 1:30pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374 "2021-10-11T13:30:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![luka.klaric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luka.klaric/32/71073_2.png) [@luka.klaric](https://discuss.elastic.co/u/luka.klaric)\
**Post date:** [October 11, 2021, 1:30pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374/1 "2021-10-11T13:30:54Z")

</div>

I am having the issue that filebeat (7.8) somehow 'ingores' my folder structure.  
This is the part of the filebeat.yml related to the folder structure:

```auto
    - type: log
      paths:
       - "/folder1/folder2/folder3/logs/app1/app1_*/*/*/*.log"
      ignore_older: 360h
      fields:
       index_suffix: "log-app1"
       location: mtc
       clustername: clustername
       stage: production
      multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

```

Is there maybe a limit to the folder tree size, because the "app1\_\*" has more than 1000 folder, and everyone of them has his own subfolders.

journalctl snippet:

```auto
...
Oct 08 08:59:53 smtca01489 filebeat[44048]: 2021-10-08T08:59:53.545+0200 INFO log/input.go:152 Configured paths: [/folder1/folder2/folder3/logs/app1/app1_*/*/*/*.log]
Oct 08 08:59:53 smtca01489 filebeat[44048]: 2021-10-08T08:59:53.545+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: %d)3335666929714024466
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200 INFO log/input.go:722 input outlet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200 ERROR log/input.go:487 Updating ignore_older state error: input outllet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200 INFO log/input.go:722 input outlet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200 ERROR log/input.go:487 Updating ignore_older state error: input outllet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200 INFO log/input.go:722 input outlet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200 ERROR log/input.go:487 Updating ignore_older state error: input outllet closed
...

```

---

<div class="post-metadata">

**Author:** ![zx8086](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zx8086/32/94917_2.png) [@zx8086](https://discuss.elastic.co/u/zx8086)\
**Post date:** [October 11, 2021, 5:10pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374/2 "2021-10-11T17:10:12Z")

</div>

Hi,

Take a look at the recursive glob pattern

> **[Log input | Filebeat Reference \[7.15\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.15/filebeat-input-log.html)**

> recursive\_glob.enabled  
> Enable expanding \*\* into recursive glob patterns. With this feature enabled, the rightmost \*\* in each path is expanded into a fixed number of glob patterns. For example: /foo/\*\* expands to /foo, /foo/_, /foo/_/\*, and so on. If enabled it expands a single \*\* into a 8-level deep \* pattern.

---

<div class="post-metadata">

**Author:** ![luka.klaric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luka.klaric/32/71073_2.png) [@luka.klaric](https://discuss.elastic.co/u/luka.klaric)\
**Post date:** [October 12, 2021, 12:55pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374/3 "2021-10-12T12:55:35Z")

</div>

On my sandbox environment, both paths are working (recursive\_glob and the normal path with wildcards). Maybe I should also mention that the files are on a network drive, but other folder structures a also harvested, so I couldn't imagine that problem is that.

Could it be that the "ignore\_older" setting causes the problem? Because, the journalctl is also mentioning this setting. And as I also mentioned, there are many files and sub-folders under this path. Are there maybe some limitations for paths?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2021, 2:55pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-harvest-folder-structure/286374/4 "2021-11-09T14:55:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.