Filebeat : how to exclude lines

Elastic Stack Beats
1

Hi,
Using ELK 7.14.0 release.
I'm trying to excludes lines from IIS access log files. I've tried several methods but It still will not work.
I've done that previously with logstash, but I prefer use a simplified architecture.

Last try was to include the following line in filebeat.yml:

processors:
   - drop_event:
     when.equals.http.request.method: OPTIONS

I'm tryng to exclude healthcheck lines like these:

2021-05-25 00:03:31 W3SVC2 prewww3 172.25.50.72 OPTIONS / - 80 - 172.25.50.123 HTTP/1.0 - - - - 200 0 0 226 22 2 -

Thanks for help,
Thomas

2

Have you tried the exclude_lines options on the input?
I'm using that since Version 5 to exclude some nonsensical lines (empty lines and separator lines) in our logs.

filebeat.inputs:
- type: log
  exclude_lines: [ "^-*$", "^$" ]
...

For your case it would probably be something like
excluse_lines: [ "OPTIONS" ]

3

Hi,
Yes, I tried this options in filebeat.yml:
exclude_lines: [ 'OPTIONS' ]

It won't work I think, because the log files path comes from iis.yml module configuration files.
the previous exclude_lines option seems to work only for log paths files from the filebeat.yml file.

4

If I read the documentation correctly it should be possible to override these module values.

Where did you define this exclude_lines?
I think it should be under the access configuration of the module like so:

- module: iis
  access:
    enabled: true
    exclude_lines: [ "^$", "OPTIONS" ]
    var.paths: ["C:/inetpub/logs/LogFiles/*/*.log"]

(see "filebeat path"\module\iis\access\config)

5

Yes I tried this in iis.yml config file that matches my config :

 access:
    enabled: true
    var.paths: [E:\LOGS\IIS\*\*\*.log]
    exclude_lines: [ "OPTIONS" ]

Unfortunately the lines are still present in elastic:

2021-08-24 08:49:31 W3SVC9 xxx xxx OPTIONS / - 8022 - 172.25.50.123 HTTP/1.0 - - - - 200 0 0 303 22 3 -
6

Sorry my example was wrong. The input level was missing (see Override input settings | Filebeat Reference [7.14] | Elastic).
So the following should be in filebeat.yml:

- module: iis
  access:
    enabled: true
    input:
      exclude_lines: [ "^$", "OPTIONS" ]
    var.paths: ["C:/inetpub/logs/LogFiles/*/*.log"]

If that isn't working i have no more ideas.

7

Hi Christian,
I've tried your exact syntax and it works fine :slight_smile: Thank you !!!

- module: iis
  # Access logs
  access:
    enabled: true
    var.paths: [E:\LOGS\IIS\*\*\*.log]
    input:
        exclude_lines: [ "^$", "OPTIONS" ]
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.