Filebeat doesn't harvest folder structure

Elastic Stack Beats
1

I am having the issue that filebeat (7.8) somehow 'ingores' my folder structure.
This is the part of the filebeat.yml related to the folder structure:

    - type: log
      paths:
       - "/folder1/folder2/folder3/logs/app1/app1_*/*/*/*.log"
      ignore_older: 360h
      fields:
       index_suffix: "log-app1"
       location: mtc
       clustername: clustername
       stage: production
      multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

Is there maybe a limit to the folder tree size, because the "app1_*" has more than 1000 folder, and everyone of them has his own subfolders.

journalctl snippet:

...
Oct 08 08:59:53 smtca01489 filebeat[44048]: 2021-10-08T08:59:53.545+0200        INFO        log/input.go:152        Configured paths: [/folder1/folder2/folder3/logs/app1/app1_*/*/*/*.log]
Oct 08 08:59:53 smtca01489 filebeat[44048]: 2021-10-08T08:59:53.545+0200        INFO        [crawler]        beater/crawler.go:141        Starting input (ID: %d)3335666929714024466
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200        INFO        log/input.go:722        input outlet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200        ERROR        log/input.go:487        Updating ignore_older state error: input outllet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200        INFO        log/input.go:722        input outlet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200        ERROR        log/input.go:487        Updating ignore_older state error: input outllet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200        INFO        log/input.go:722        input outlet closed
Oct 08 09:00:03 smtca01489 filebeat[44048]: 2021-10-08T09:00:03.328+0200        ERROR        log/input.go:487        Updating ignore_older state error: input outllet closed
...
2

Hi,

Take a look at the recursive glob pattern

recursive_glob.enabled
Enable expanding ** into recursive glob patterns. With this feature enabled, the rightmost ** in each path is expanded into a fixed number of glob patterns. For example: /foo/** expands to /foo, /foo/, /foo//*, and so on. If enabled it expands a single ** into a 8-level deep * pattern.

3

On my sandbox environment, both paths are working (recursive_glob and the normal path with wildcards). Maybe I should also mention that the files are on a network drive, but other folder structures a also harvested, so I couldn't imagine that problem is that.

Could it be that the "ignore_older" setting causes the problem? Because, the journalctl is also mentioning this setting. And as I also mentioned, there are many files and sub-folders under this path. Are there maybe some limitations for paths?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.