Filebeat Bug: exclude_lines

[Peter_Boos]

BUG report
filebeat version : filebeat-7.10.1-2021.01.13-000001
problem : The exclude_lines option in filebeat.yml does not work.

Example item to be filtered out :

2021-01-27T02:08:31.775-0500#011INFO#011log/harvester.go:302#011Harvester started for file: /var/log/syslog
2021-01-27T02:08:22.758-0500#011INFO#011log/harvester.go:333#011File is inactive: /var/log/syslog. Closing because close_inactive of 5m0s reached.

Tested all kinds regex syntax filters none works

# as : is used in regex i use . instead
exclude_lines: ['\bharvester/.go.333|\bharvester/.go.302']
exclude_lines: ['\bgo.333','\bgo.303','^DBG']
exclude_lines: ['.*go.333.*', '.*go.302.*', '^DBG']

\b first word pattern matching
.* any characters till pattern
. any sungle character (replacing the : )

filebeat.inputs
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  exlude_lines :      # see all the above attempts none work

[Peter_Boos]

While still a bug, and please developers explain exclude_lines better
I found an unreliable? work around that works for this case
I still would prefer the other syntax using regex, but it doesnt work.
I tried with when: regexp message:

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_event:
      when:
        or:
          - contains.message: "harvester.go:302"
          - contains.message: "harvester.go:333"                                                                                                                

[shaunak (Shaunak Kashyap)]

302 and 333 are line numbers in the harvester.go file from where the log messages are being generated. It seems unreliable to refer to specific line numbers in the exclude pattern. In some future release these line numbers could change.

Without using specific line numbers could you explain what types of lines you are trying to exclude? Maybe there is a better expression we can come up with.

Shaunak