Filebeat exclude line not working :(

[funtimes]

Hello all! I've tried doing the best I can and research on my issue, however everything i've tried and research doesn't seem to work

Was hoping somebody on here might be able to help me out.

I have a very simple webserver(remote) using nginx and filebeats(6.1) to ship the logs directly to ES/Kibana(6.1) box in my network.

The logs are getting there just fine, I just simply want to reduce some of the logs that are sent, as they are junk/filler logs that I do not want to parse/store in ES.

Below is an example of the nginx line i want to exclude

xxx.xxx.xxx.xxx - - [27/Dec/2017:05:57:06 -0500] "GET /feedback/user HTTP/1.1" 200 824 "https://website/page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"

Below is my nginx.yml config

- module: nginx
  access:
    enabled: true
    var.paths:
      - /var/log/nginx/access.log
    exclude_lines: ['feedback']
  error:
    enabled: true
    var.paths:
      - /var/log/nginx/error.log

Is there something that I am missing?

[robscott27]

I ran into this same issue last week and never got a response, but I did find a work around. It doesn't hold through package updates though since the files get overwritten. Not sure if this will help you, but here's the topic I opened:

Exclude_lines regex isn't working

I'm trying to exclude lines from apache log that contain /server-status?auto= within the line. It's a standard apache combined log file. exclude_lines expressions I've tried: ['(?i:/server-status?auto=)'] ['.server-status.'] ['GET /server-status'] as well as a few other iterations I can't remember along the way. I even tried ['.'] and ['*'] in order to trigger excluding EVERYTHING in the log just to make sure it was actually processing but neither of those had any effect and the logs still…

[funtimes]

@robscott27 You are awesome!!!!!

I followed your example for the nginx config file at

/usr/share/filebeat/module/nginx/access/config/nginx-access.yml

Added this last line to the bottom

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
exclude_lines: ['.*feedback.*']

And now it works as expected!!!! Thank you so much!! Wish I would have found that post earlier! Have been beating my head against the wall for days.

[robscott27]

Just remember that if/when you update the filebeat package, those changes
will not persist. They will be overwritten when the package is updated. I
can confirm this is the case when I updated from 6.1.0 to 6.1.1.

As mentioned, this is not the best way to get the exclusions working and I
never heard from anyone in that topic post as to why those config files
aren't being read/processed. Good to know it's not just me running into
this!

[ruflin]

The exclude_lines should also be prefixed with var I think, then it should work.

[funtimes]

elastic/beats/blob/master/filebeat/filebeat.reference.yml#L297

So is line 297 in the file filebeat.reference.yml incorrect?

#exclude_lines: ['^DBG']

[ruflin]

My above comment should have said prefixed by prospector and not var.

About the file you linked: There is a difference between configuring a prospector and a module. A module is using a prospector but has some predefined config options for the specific module. So from the module you need to use the prospector prefix to access the prospector config options.