Cant get regex to exclude line to work

Elastic Stack Beats
1

Newbie to ELK Stack and Filebeat here. Managed to get log shipping from my linux hosts working fine and have 2x prospectors for my syslogs and secure logs as going to try filtering them in logstash differently.

Anyway, I have a service running on all servers (NRPE for Nagios) which puts syslog entries every couple of minutes which I don't want to include, but I cannot seem to get filebeat to exclude them. This is what I currently have.

- input_type: log
  paths:
    - /var/log/messages*
    - /var/log/syslog*
  exclude_files: [".gz$"]
  exclude_lines: ['.*nrpe.*', '.*Nrpe.*']
  multiline:
    pattern: "^\\s"
    match: after
  fields:
      logtype: syslog_data

I've also tried

exclude_lines: ['[nN]rpe']
exclude_lines: [(?i)nrpe]

Any ideas what I'm doing wrong?

2

Can you paste also an example log line created by NRPE?

3

Here you go -

Aug  2 12:16:13 thisuser xinetd[16389]: START: nrpe pid=40975 from=::ffff:10.10.10.4
Aug  2 12:16:13 thisuser xinetd[16389]: EXIT: nrpe status=0 pid=40975 duration=0(sec)
Aug  2 12:17:09 thisuser xinetd[16389]: START: nrpe pid=41024 from=::ffff:10.10.10.4
Aug  2 12:17:09 thisuser xinetd[16389]: EXIT: nrpe status=0 pid=41024 duration=0(sec)

I was trying to do it on case insensitive just in case it ever changes.

4

Seem to work fine in my tests with simply: exclude_lines: ['nrpe']

Can you try without the multiline config? I worry about the use of " there. Single quotes are less problematic when it comes to regexps.

5

Really odd. I tried with just the 'nrpe' as you suggested above and still doesn't pull it out. So I commented out the multiline sections and it still doesn't strip it out. Not sure what is happening.

I only had the multiline in as that was the default config for syslog from the filebeat/logstash side.

6

Correction - I have got it to work. I'm not sure why but on my test lab it wasn't filtering it out, but when I tried this with one of my live systems with just a single agent it filtered it out OK! I then tried with the below and that also worked filtering out multiple entires.

exclude_lines: ['nrpe', 'bamboo', 'java.lang']

Much appreciated! Have also knocked up a simple Ansible playbook to push this out and allow me to adjust the config on the fly :slight_smile:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.