# Filebeat exclude\_files is not working as expected
**URL:** https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995
**Category:** Beats
**Tags:** filebeat
**Created:** [May 31, 2022, 5:27am UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995 "2022-05-31T05:27:08Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@omeryosef](https://discuss.elastic.co/u/omeryosef)
#### Post date: [May 31, 2022, 5:27am UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/1 "2022-05-31T05:27:08Z")
Hi everyone,
I have the following structure of directories and I am trying to avoid duplications by excluding "current" dir:
```auto
# ls -l
total 12
drwxrwxr-x 11 node node 4096 May 25 10:42 3.123.0
drwxrwxr-x 11 node node 4096 May 30 10:16 3.124.0
lrwxrwxrwx 1 node node 12 May 30 10:16 current -> /ver/3.124.0
drwxrwxr-x 2 node node 4096 Jun 1 2020 logs
```
This is my configuration filebeat yml input:
```auto
############################## Inputs ##################################
filebeat.inputs:
- type: filestream
enabled: true
paths:
- "/ver/*/logs/*.log"
exclude_files: '/^\/ver\/(current)\/logs.+.log/gm'
```
the regexp was verified with [https://regex101.com/](https://regex101.com/)
However I still see logs from current dir arrive to Kibana.
Any help would be much appreciated.
Thanks,
Omer.
---
### Author:  [@ibra\_013](https://discuss.elastic.co/u/ibra_013)
#### Post date: [June 1, 2022, 10:05am UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/2 "2022-06-01T10:05:32Z")
Hi @omeryosef Welcome to the community.
you are using log input syntax in filestream input, it will not work,
```auto
filebeat.inputs:
- type: filestream
...
prospector.scanner.exclude_files: ['\.gz$']
```
> **[filestream input | Filebeat Reference \[8.2\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-exclude-files)**
---
### Author:  [@omeryosef](https://discuss.elastic.co/u/omeryosef)
#### Post date: [June 1, 2022, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/3 "2022-06-01T13:29:23Z")
Hi,
Thanks for your reply!
Regarding the type, I am working with filebeat 7.16 so I see the log type is deprecated in the docs so I am using filestream and assume that the rest of the syntax is the same:
> **[Log input | Filebeat Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.16/filebeat-input-log.html)**
Anyway,
using the filestream type with these options didn't work as well (I still see logs from current folder).
Tried both:
prospector.scanner.exclude\_files: ['/^/ver/(current)/logs.+.log/g']
prospector.scanner.exclude\_files: ['/^/ver/(current)/logs.+.log/gm']
```auto
############################## Inputs ##################################
filebeat.inputs:
- type: filestream
enabled: true
paths:
- "/ver/*/logs/*.log"
prospector.scanner.exclude_files: ['/^\/ver\/(current)\/logs.+.log/g']
tags: ["portal","node-5","node-6"]
```
Not sure what I am missing...
---
### Author:  [@ibra\_013](https://discuss.elastic.co/u/ibra_013)
#### Post date: [June 1, 2022, 2:15pm UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/4 "2022-06-01T14:15:01Z")
Hi @omeryosef
`current` file or folder is a symlink to `/ver/3.124.0` and from the listing shared
`/ver/3.124.0` is the current folder or file and is not excluded. right ?
> [@omeryosef](#):
>
> ```auto
> # ls -l
> total 12
> drwxrwxr-x 11 node node 4096 May 25 10:42 3.123.0
> drwxrwxr-x 11 node node 4096 May 30 10:16 3.124.0
> lrwxrwxrwx 1 node node 12 May 30 10:16 current -> /ver/3.124.0
> drwxrwxr-x 2 node node 4096 Jun 1 2020 logs
>
> ```
---
### Author:  [@omeryosef](https://discuss.elastic.co/u/omeryosef)
#### Post date: [June 1, 2022, 5:42pm UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/5 "2022-06-01T17:42:29Z")
right, and I see that the file is current (duplicate events).
BTW, it didn't work with log type as well with this config:
```auto
############################## Inputs ##################################
filebeat.inputs:
- type: log
enabled: true
paths:
- "/ver/*/logs/*.log"
exclude_files: '/^\/ver\/(current)\/logs.+.log/gm'
```
And I have validate the regex with [https://regex101.com/](https://regex101.com/) as I mentioned.
---
### Author:  [@ibra\_013](https://discuss.elastic.co/u/ibra_013)
#### Post date: [June 2, 2022, 8:59am UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/6 "2022-06-02T08:59:43Z")
Hi @omeryosef
exclude\_files is a list and by default, the symlinks are disabled.
```auto
# If symlinks is enabled, symlinks are opened and harvested. The harvester is opening the
# original for harvesting but will report the symlink name as source.
#symlinks: false
```
> [@omeryosef](#):
>
> ```auto
> paths:
> - "/ver/*/logs/*.log"
> exclude_files: ['/^\/ver\/(current)\/logs.+.log/gm']
>
> ```
```auto
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
```
> **[filebeat.reference.yml | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html)**
---
### Author:  [@omeryosef](https://discuss.elastic.co/u/omeryosef)
#### Post date: [June 2, 2022, 9:29am UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/7 "2022-06-02T09:29:58Z")
I see,
thanks a lot for putting my attention to this.
I need the indication of the original dir (in the above ls -l output it would be 3.124.0).
In order to do that I am using the following configuration:
```auto
############################## Inputs ##################################
filebeat.inputs:
- type: log
enabled: true
paths:
- "/ver/*/logs/*.log"
exclude_files:
- '^\/ver\/.+(current)\/logs.+.log'
multiline.pattern: '^[[:upper:]]|^\[[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{2}\/[0-9]{2}\/[0-9]{4}|^[0-9]{2}-[[:alpha:]]{3}-[0-9]{4}|^\{'
multiline.negate: true
multiline.match: after
```
currently I can see logs from the 3.124.0 and I don't see logs from "current".
is this configuration can cause data loss? or is is the right way to have all data with indication from the original dir (3.124.0) and not the symlink (current)?
Thank a lot!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 30, 2022, 11:30am UTC](https://discuss.elastic.co/t/filebeat-exclude-files-is-not-working-as-expected/305995/8 "2022-06-30T11:30:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.