Exclude_lines regex isn't working

[robscott27]

I'm trying to exclude lines from apache log that contain /server-status?auto= within the line. It's a standard apache combined log file.

exclude_lines expressions I've tried:

['(?i:/server-status?auto=)']
['.server-status.']
['GET /server-status']

as well as a few other iterations I can't remember along the way.

I even tried ['.'] and ['*'] in order to trigger excluding EVERYTHING in the log just to make sure it was actually processing but neither of those had any effect and the logs still were being picked up/inserted into ES.

Like several other posters, it works in Go, but when put into the apache2.yml file, it doesn't. I've also got exclude_files: [".gz$"] and in the debug logs, I can clearly see it excluding those files.

Example of a log line that I'd like to exclude:

example.site.com:80 192.168.0.1 - - [20/Dec/2017:10:18:37 -0500] "GET /server-status?auto= HTTP/1.1" 301 522 "-" "Go-http-client/1.1"

Any ideas what I'm missing?

Thanks in advance!

[robscott27]

I figured this one out. I noticed that even though I had commented out the exclude_files line, it was still excluding the .gz log files, even though it shouldn't. So I went digging.

Turns out, there's an access.yml file located here:
/usr/share/filebeat/module/apache2/access/config

which still had the exclude_files option set to ignore the .gz files. Below that line, I added:
exclude_lines: ['.(?:server-status).']

and restarted the service. Watched the log file, and bingo, lines are being dropped now.

Not sure why it's ignoring processing the apache2.yml file in /etc/filebeat/modules.d/ though.

So if anyone else is having trouble with the exclude_lines: option in apache.2yml, try checking that other file location listed above.

Not sure that change will survive a package update, but it's a small one and is easy to put back if need be.

[ruflin]

If you want to overwrite config options for the prospector from modules.d you have to use you have to prefix it with var: https://www.elastic.co/guide/en/beats/filebeat/current/specify-variable-settings.html

[robscott27]

So, by that reasoning, if I wanted to modify the exclude_lines config
option in the apache2.yml access section - which isn't set to anything in
the modules.d location - I would add a line:

var.exclude_lines: [pattern]

However, in practice the lines are not excluded and still show up in ES
when using var.exclude_lines in /etc/filebeat/modules.d/apache2.yml. Also,
from what I can tell on this page:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-apache2.html

the only variable settings listed are var.paths for both the access and
error areas of the module.

[ruflin]

I'm sorry my comment above was wrong. I confused the var configs with the prospector configs. As exclude_lines is a prospector option, you need to add it under the prospector namespace: https://www.elastic.co/guide/en/beats/filebeat/current/advanced-settings.html