Exclude file pattern not working correctly

[brjisc (Ben Ryan)]

Hi,
With this configuration:

filebeat.inputs:

• type: log

  enabled: true

  paths:

  • /tmp/cssd/*
    exclude_files: ['^error.+$','^.+error$','^bar.+','^.+error.+$','^.+.log.1','.*.gz']

Creating these files in the /tmp/cssd directory:

touch error-foo.log
touch foo-error.log
touch foo-error
touch foo.log.1
touch foo.log.gz
touch foo.log

touch error-bar.log
touch bar-error.log
touch bar-error
touch bar.log.1
touch bar.log.gz
touch bar.log

Gives the following log output:

2018-08-24T07:31:53.659+0100 INFO registrar/registrar.go:124 States Loaded from registrar: 3
2018-08-24T07:31:53.659+0100 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-24T07:31:53.659+0100 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-08-24T07:31:53.661+0100 INFO log/input.go:118 Configured paths: [/tmp/cssd/*]
2018-08-24T07:31:53.661+0100 INFO input/input.go:88 Starting input of type: log; ID: 1762196086849029994
2018-08-24T07:31:53.661+0100 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-08-24T07:31:53.661+0100 INFO cfgfile/reload.go:122 Config reloader started
2018-08-24T07:31:53.661+0100 INFO cfgfile/reload.go:214 Loading of config files completed.
2018-08-24T07:32:03.662+0100 INFO log/harvester.go:228 Harvester started for file: /tmp/cssd/bar.log
2018-08-24T07:32:03.662+0100 INFO log/harvester.go:228 Harvester started for file: /tmp/cssd/foo.log

I cannot work out why Filebeat is starting a Harvester for the bar.log

Ben

[pierhugues (Pier-Hugues Pellerin)]

Hello @brjisc, Theexclude_files options need to operate on the full path and not only of the filename. The regular expressions that you have defined above will never match a complete path.

exclude_files: ['^error.+$','^.+error$','^bar.+','^.+error.+$','^.+.log.1','.*.gz']

You are matching ^bar.+ which would match a string beginning with the world bar but not the following string /tmp/cssd/bar.log