Filebeat exclude lines not working

Elastic Stack Beats
1

Filebeats
does not exclude the lines as I expected

my configuration is

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
 exclude_lines: ['.js','.png','.svg','.json','.txt']
 #also i try
 exclude_lines: ['.*\.(js|svg|png|json|txt).*']

in /etc/filebeat/filebeat.yml

but keep creating documents in ES

Captura%20de%20pantalla%20de%202018-07-13%2000-04-46

1 Like
2

Hi @HerberthObregon and welcome :slight_smile:

Indeed your regular expression looks fine, but are these logs being read from /var/log/*.log? It looks like you are using the nginx module. Modules include their own prospector configurations, so your filebeat is probably not using this exclude_lines setting for the nginx logs.

Something you can try is to add a drop_event processor, that allows you to filter on the already parsed log lines, in your case I guess it'd be something like this:

processors:
  drop_event.when.regexp:
    nginx.access.url: '.*\.(js|svg|png|json|txt)$'
3

There are several folders but among them is that of nginx and there is actually the log there

y try add this to end of the file /etc/filebeat/filebeat.yml

then I run sudo service filebeat restart

I drop filebeat-* Index in ES

I go to Kibana and check but still create docs with .js files logs :frowning:

I have already searched the internet and I have not found how to send this behavior, the reason for this is that I only want to monitor my API and not my static files
You are my last hope to achieve this :frowning:

4

Could you also paste the configuration of the nginx module?

What are the paths of your nginx log files?

5

my logs is like a

0.0.0.0 - - [14/Jul/2018:06:27:41 +0000] "GET /static/js/c2log.js HTTP/1.1" 200 1675 "https://www.example.com/" "Mozilla/5.0 (Linux; Android 5.0.1; SM-N915V Build/LRX22C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36"
0.0.0.0 - - [14/Jul/2018:06:27:41 +0000] "GET /static/js/rebound.js HTTP/1.1" 200 16594 "https://www.example.com/" "Mozilla/5.0 (Linux; Android 5.0.1; SM-N915V Build/LRX22C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36"
0.0.0.0 - - [14/Jul/2018:06:27:41 +0000] "GET /static/js/qrcode.min.js HTTP/1.1" 200 20413 "https://www.example.com/" "Mozilla/5.0 (Linux; Android 5.0.1; SM-N915V Build/LRX22C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36"
0.0.0.0 - - [14/Jul/2018:06:27:41 +0000] "GET /static/values/world.js HTTP/1.1" 200 14691 "https://www.example.com/" "Mozilla/5.0 (Linux; Android 5.0.1; SM-N915V Build/LRX22C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36"
0.0.0.0 - - [14/Jul/2018:06:27:41 +0000] "GET /static/values/countries.js HTTP/1.1" 200 296755 "https://www.example.com/" "Mozilla/5.0 (Linux; Android 5.0.1; SM-N915V Build/LRX22C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36"

and my config in /usr/share/filebeat/module/nginx/access/config/nginx-access.yml

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.