Filtering processors do not work for syslog

Elastic Stack Beats
1

Hi. Using ES 6.3.0 and Filebeat 6.3.2. Try to configure filtering to exclude some lines from the syslog file.
I tried processors and/or "exclude_lines" but none of them are working. Since i can see in debug level some lines about excluding lines or filtering process, I guess this has something to do with another thing. I suspect Ingest Pipelines and/or Time conversion.

I saw in the debug log that a line is filtered out correctly based on the exclude_lines directive (drop line ...) but it still sends to Kibana few milliseconds later...

Question: If i admit that filtering using processors or exclude_lines is working:

  • do you think UTC time conversion can induces such behavior ?
  • what else may i test to sort this out ?

many thanks in advance.

francois

2

Could you please share your debug logs and configuration?

3

Hi Noémi,
Thx for your answer.
My Config:

...

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  scan_frequency: 30s

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages
    # - /var/log/secure
    - /var/log/audit/audit.log
    #- c:\programdata\elasticsearch\logs\*

   processors:
    - add_locale: 
        format: abbreviation

#    - drop_event:
#        when.not.regexp.system.syslog.message: '^(warning|critical|error)*'
    
#    - drop_event:
#        when.not.regexp.audit.log.record_type: 'USER_LOGIN*'

#    - drop_event:
#        when.contains.system.syslog.program: "nrpe"

#    - drop_event:
#        when:
#           regexp:
#              message: "^Warning:"

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  exclude_lines: ['.*nrpe.*']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']
...

and my debug log (partial)...

2018-08-14T08:44:40.812+0200	DEBUG	[harvester]	log/harvester.go:385	Drop line as it does match one of the exclude patternsAug 14 08:44:35 rh749elk9 nrpe[23040]: CONN_CHECK_PEER: checking if host is allowed: 10.166.1.42 port 37524
2018-08-14T08:44:40.812+0200	DEBUG	[harvester]	log/harvester.go:385	Drop line as it does match one of the exclude patternsAug 14 08:44:35 rh749elk9 nrpe[23040]: is_an_allowed_host (AF_INET): is host >10.166.1.42< an allowed host >10.166.1.42<
2018-08-14T08:44:40.812+0200	DEBUG	[harvester]	log/harvester.go:385	Drop line as it does match one of the exclude patternsAug 14 08:44:35 rh749elk9 nrpe[23040]: is_an_allowed_host (AF_INET): is host >10.166.1.42< an allowed host >10.166.1.42<
2018-08-14T08:44:40.812+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-08-14T08:44:40.812+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-08-14T08:44:40.812+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-08-14T08:44:40.812+0200	DEBUG	[registrar]	registrar/registrar.go:328	Processing 1 events
2018-08-14T08:44:40.812+0200	DEBUG	[registrar]	registrar/registrar.go:298	Registrar state updates processed. Count: 1
...
2018-08-14T08:44:40.853+0200	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-08-14T06:44:40.852Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.2",
    "pipeline": "filebeat-6.3.2-system-syslog-pipeline"
  },
  "offset": 8062102,
  "message": "Aug 14 08:44:35 rh749elk9 nrpe[23040]: CONN_CHECK_PEER: checking if host is allowed: 10.166.1.42 port 37524",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "fileset": {
    "name": "syslog",
    "module": "system"
  },
  "beat": {
    "timezone": "+02:00",
    "name": "rh749elk9.ceti.etat-ge.ch",
    "hostname": "rh749elk9.ceti.etat-ge.ch",
    "version": "6.3.2"
  },
  "host": {
    "name": "rh749elk9.ceti.etat-ge.ch"
  },
  "source": "/var/log/messages"
}
2018-08-14T08:44:40.853+0200	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-08-14T06:44:40.852Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.2",
    "pipeline": "filebeat-6.3.2-system-syslog-pipeline"
  },
  "host": {
    "name": "rh749elk9.ceti.etat-ge.ch"
  },
  "source": "/var/log/messages",
  "offset": 8062210,
  "message": "Aug 14 08:44:35 rh749elk9 nrpe[23040]: is_an_allowed_host (AF_INET): is host \u003e10.166.1.42\u003c an allowed host \u003e10.166.1.42\u003c",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "fileset": {
    "name": "syslog",
    "module": "system"
  },
  "beat": {
    "hostname": "rh749elk9.ceti.etat-ge.ch",
    "version": "6.3.2",
    "name": "rh749elk9.ceti.etat-ge.ch",
    "timezone": "+02:00"
  }
}
...

Hope it helps
thx

4

Oh, so are using log input? Or do you want to use the system/syslog module of Filebeat?

5

hum ... maybe there is a misunderstanding from me.

I am using under modules.d the system.yml with syslog inside for the messages file.
But i was not able to declare any processors in this section, it never worked.

Additionnaly, as far as i understood, there is no exclude_lines in the file system.yml. So you mean that the log input can not work because later on the system/syslog takes care of the lines, right ?

In this case, do you have any examples on how to declare filtering processors in the system.yml ?

I hope i am clear enough. Thx

6

Now I understand everything. My bad. Before you shared your configuration and debug logs I thought you want to use the syslog input (https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-input-syslog.html).

It's not possible to use inputs and modules in the same Filebeat. If you have any module enabled, Filebeat is only going to collect and process files set in the module. But in your case it's not a problem. Enabling the module in system.yml is sufficient, because you can add processors to the top level configuration of Filebeat. So there is no need to use inputs.

processors:
- add_locale: 
    format: abbreviation
7

Ok Noèmi. Now i understand also my mistake.
So, let's assume we want to use only syslog olude, no input log.
I would like to specify some timing parameters and some processors.

In all my tests i had some trouble. The problem is that all examples i can find, nerver show any context where to put the directives. Could you please tell me if my system.yml is correct in your point of view ?

- module: system

#    Syslog
  syslog:
    enabled: true

    processors:
      - drop_event:
          when.contains.system.syslog.program: "nrpe"

    exclude_lines: ['.*nrpe.*']

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    var.paths : ["/var/log/messages"]

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

It seems that it is accepted by filebeat but nothing appends (no filtering). Also i would like to add some timing directives:

scan_frequency: 1m
close_inactive: 5m
events: 4096
flush.min_event: 500

Is it possible to add these settings ? If yes, where exactly ?

Anyhow, thank you very much for your great help ... I appreciate

Francois

8

You need to add processors directive to filebeat.yml, not to system.yml.

9

Thx Noémi ... so i guess on the top of the file, not in the input section, right ? many thx

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.