# Can you exclude subfolder from being crawled completely in filebeat?

**URL:** <https://discuss.elastic.co/t/can-you-exclude-subfolder-from-being-crawled-completely-in-filebeat/381127>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 19, 2025, 8:25am UTC](https://discuss.elastic.co/t/can-you-exclude-subfolder-from-being-crawled-completely-in-filebeat/381127 "2025-08-19T08:25:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![WERTYASDFGH](https://avatars.discourse-cdn.com/v4/letter/w/ccd318/32.png) [@WERTYASDFGH](https://discuss.elastic.co/u/WERTYASDFGH)\
**Post date:** [August 19, 2025, 8:25am UTC](https://discuss.elastic.co/t/can-you-exclude-subfolder-from-being-crawled-completely-in-filebeat/381127/1 "2025-08-19T08:25:19Z")

</div>

Hello, I’m trying to collect logs from certain folder using filebeat 8.18.3. All the logs are in subfolders of this folder. However in the same directory as logs there 2 folders that contains few millions of xml files that are not logs(not my decision, I have no idea why it’s there). So when I run filebeat, it starts using more and more ram and crashes.

Here is part of config:

```auto
filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: "some-id"

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /some/path/logs/**/*.txt

  prospector.scanner.exclude_files: [
    '^\/some\/path\/logs\/swap_files\/.+',
    '^\/some\/path\/logs\/export_files\/.+'
  ]

  pipeline: "some-pipeline"
  parsers:
  - multiline:
      pattern: '^\[\d{2}:\d{2}:\d{2}\.\d{3} [A-Z]{3}\]'
      negate: true
      match: after

```

I tried also different exclude files:  
``  
` prospector.scanner.exclude_files: [ `  
` '^\/some\/path\/logs\/swap_files\/',`  
` '^\/some\/path\/logs\/export_files\/']`  
```

```auto
  prospector.scanner.exclude_files: [
    '^/some/path/logs/swap_files/',
    '^/some/path/logs/export_files/'
  ]

```

However none of it seems to work. From what I understand it still tries match regex on all of files names so it doesn’t help to solve my problem. For now I just specify multiple paths for every subfolder except those 2 but this doesn’t seem like scalable solution.

---

<div class="post-metadata">

**Author:** ![Tortoise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tortoise/32/147587_2.png) [@Tortoise](https://discuss.elastic.co/u/Tortoise)\
**Post date:** [August 19, 2025, 9:56am UTC](https://discuss.elastic.co/t/can-you-exclude-subfolder-from-being-crawled-completely-in-filebeat/381127/2 "2025-08-19T09:56:08Z")

</div>

Hello @WERTYASDFGH

Welcome to the community!!

As per below blog try without using ^ as you are providing absolute path :

> [@Regex not worked with filebeat](https://discuss.elastic.co/t/regex-not-worked-with-filebeat/351901):
>
> Hi Dear, I'm trying to exclude any files starting with gc but below regex did not work and regex is verified, myfilebeat version is :8.3.2 filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. # filestream is an input for collecting log messages from files. - type: filestream # Unique ID among all inputs, an ID is required. id: my-filestream-id # …

Thanks!!

---

<div class="post-metadata">

**Author:** ![WERTYASDFGH](https://avatars.discourse-cdn.com/v4/letter/w/ccd318/32.png) [@WERTYASDFGH](https://discuss.elastic.co/u/WERTYASDFGH)\
**Post date:** [August 19, 2025, 10:52am UTC](https://discuss.elastic.co/t/can-you-exclude-subfolder-from-being-crawled-completely-in-filebeat/381127/3 "2025-08-19T10:52:56Z")

</div>

I tried, it doesnt change anything. Probably because it still scans the folders I want to omit, and because it matches regex of full paths of files and not folders. So in my case it scans concurently names of millions of files which causes extreme spike in ram and crashing. At least that’s my theory