# Filtering processors do not work for syslog

**URL:** <https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 14, 2018, 8:43am UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301 "2018-08-14T08:43:49Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fpernet](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@Fpernet](https://discuss.elastic.co/u/Fpernet)\
**Post date:** [August 14, 2018, 8:43am UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/1 "2018-08-14T08:43:49Z")

</div>

Hi. Using ES 6.3.0 and Filebeat 6.3.2. Try to configure filtering to exclude some lines from the syslog file.  
I tried processors and/or "exclude\_lines" but none of them are working. Since i can see in debug level some lines about excluding lines or filtering process, I guess this has something to do with another thing. I suspect Ingest Pipelines and/or Time conversion.

I saw in the debug log that a line is filtered out correctly based on the exclude\_lines directive (drop line ...) but it still sends to Kibana few milliseconds later...

Question: If i admit that filtering using processors or exclude\_lines is working:

- do you think UTC time conversion can induces such behavior ?
- what else may i test to sort this out ?

many thanks in advance.

francois

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 14, 2018, 8:49am UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/2 "2018-08-14T08:49:13Z")

</div>

Could you please share your debug logs and configuration?

---

<div class="post-metadata">

**Author:** ![Fpernet](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@Fpernet](https://discuss.elastic.co/u/Fpernet)\
**Post date:** [August 14, 2018, 9:44am UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/3 "2018-08-14T09:44:56Z")

</div>

Hi Noémi,  
Thx for your answer.  
My Config:

```
...

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  scan_frequency: 30s

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages
    # - /var/log/secure
    - /var/log/audit/audit.log
    #- c:\programdata\elasticsearch\logs\*

   processors:
    - add_locale: 
        format: abbreviation

# - drop_event:
# when.not.regexp.system.syslog.message: '^(warning|critical|error)*'
    
# - drop_event:
# when.not.regexp.audit.log.record_type: 'USER_LOGIN*'

# - drop_event:
# when.contains.system.syslog.program: "nrpe"

# - drop_event:
# when:
# regexp:
# message: "^Warning:"

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  exclude_lines: ['.*nrpe.*']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']
...

```

and my debug log (partial)...

```
2018-08-14T08:44:40.812+0200	DEBUG	[harvester]	log/harvester.go:385	Drop line as it does match one of the exclude patternsAug 14 08:44:35 rh749elk9 nrpe[23040]: CONN_CHECK_PEER: checking if host is allowed: 10.166.1.42 port 37524
2018-08-14T08:44:40.812+0200	DEBUG	[harvester]	log/harvester.go:385	Drop line as it does match one of the exclude patternsAug 14 08:44:35 rh749elk9 nrpe[23040]: is_an_allowed_host (AF_INET): is host >10.166.1.42< an allowed host >10.166.1.42<
2018-08-14T08:44:40.812+0200	DEBUG	[harvester]	log/harvester.go:385	Drop line as it does match one of the exclude patternsAug 14 08:44:35 rh749elk9 nrpe[23040]: is_an_allowed_host (AF_INET): is host >10.166.1.42< an allowed host >10.166.1.42<
2018-08-14T08:44:40.812+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-08-14T08:44:40.812+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-08-14T08:44:40.812+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-08-14T08:44:40.812+0200	DEBUG	[registrar]	registrar/registrar.go:328	Processing 1 events
2018-08-14T08:44:40.812+0200	DEBUG	[registrar]	registrar/registrar.go:298	Registrar state updates processed. Count: 1
...
2018-08-14T08:44:40.853+0200	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-08-14T06:44:40.852Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.2",
    "pipeline": "filebeat-6.3.2-system-syslog-pipeline"
  },
  "offset": 8062102,
  "message": "Aug 14 08:44:35 rh749elk9 nrpe[23040]: CONN_CHECK_PEER: checking if host is allowed: 10.166.1.42 port 37524",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "fileset": {
    "name": "syslog",
    "module": "system"
  },
  "beat": {
    "timezone": "+02:00",
    "name": "rh749elk9.ceti.etat-ge.ch",
    "hostname": "rh749elk9.ceti.etat-ge.ch",
    "version": "6.3.2"
  },
  "host": {
    "name": "rh749elk9.ceti.etat-ge.ch"
  },
  "source": "/var/log/messages"
}
2018-08-14T08:44:40.853+0200	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-08-14T06:44:40.852Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.2",
    "pipeline": "filebeat-6.3.2-system-syslog-pipeline"
  },
  "host": {
    "name": "rh749elk9.ceti.etat-ge.ch"
  },
  "source": "/var/log/messages",
  "offset": 8062210,
  "message": "Aug 14 08:44:35 rh749elk9 nrpe[23040]: is_an_allowed_host (AF_INET): is host \u003e10.166.1.42\u003c an allowed host \u003e10.166.1.42\u003c",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "fileset": {
    "name": "syslog",
    "module": "system"
  },
  "beat": {
    "hostname": "rh749elk9.ceti.etat-ge.ch",
    "version": "6.3.2",
    "name": "rh749elk9.ceti.etat-ge.ch",
    "timezone": "+02:00"
  }
}
...

```

Hope it helps  
thx

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 14, 2018, 11:37am UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/4 "2018-08-14T11:37:18Z")

</div>

Oh, so are using `log` input? Or do you want to use the `system/syslog` module of Filebeat?

---

<div class="post-metadata">

**Author:** ![Fpernet](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@Fpernet](https://discuss.elastic.co/u/Fpernet)\
**Post date:** [August 14, 2018, 11:58am UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/5 "2018-08-14T11:58:12Z")

</div>

hum ... maybe there is a misunderstanding from me.

I am using under modules.d the system.yml with syslog inside for the messages file.  
But i was not able to declare any processors in this section, it never worked.

Additionnaly, as far as i understood, there is no exclude\_lines in the file system.yml. So you mean that the log input can not work because later on the system/syslog takes care of the lines, right ?

In this case, do you have any examples on how to declare filtering processors in the system.yml ?

I hope i am clear enough. Thx

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 14, 2018, 1:39pm UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/6 "2018-08-14T13:39:13Z")

</div>

Now I understand everything. My bad. Before you shared your configuration and debug logs I thought you want to use the syslog input ([https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-input-syslog.html](https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-input-syslog.html)).

It's not possible to use inputs and modules in the same Filebeat. If you have any module enabled, Filebeat is only going to collect and process files set in the module. But in your case it's not a problem. Enabling the module in `system.yml` is sufficient, because you can add processors to the top level configuration of Filebeat. So there is no need to use inputs.

```auto
processors:
- add_locale: 
    format: abbreviation

```

---

<div class="post-metadata">

**Author:** ![Fpernet](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@Fpernet](https://discuss.elastic.co/u/Fpernet)\
**Post date:** [August 14, 2018, 2:33pm UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/7 "2018-08-14T14:33:15Z")

</div>

Ok Noèmi. Now i understand also my mistake.  
So, let's assume we want to use only syslog olude, no input log.  
I would like to specify some timing parameters and some processors.

In all my tests i had some trouble. The problem is that all examples i can find, nerver show any context where to put the directives. Could you please tell me if my system.yml is correct in your point of view ?

```
- module: system

# Syslog
  syslog:
    enabled: true

    processors:
      - drop_event:
          when.contains.system.syslog.program: "nrpe"

    exclude_lines: ['.*nrpe.*']

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    var.paths : ["/var/log/messages"]

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

```

It seems that it is accepted by filebeat but nothing appends (no filtering). Also i would like to add some timing directives:

```
scan_frequency: 1m
close_inactive: 5m
events: 4096
flush.min_event: 500

```

Is it possible to add these settings ? If yes, where exactly ?

Anyhow, thank you very much for your great help ... I appreciate

Francois

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 15, 2018, 12:31pm UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/8 "2018-08-15T12:31:26Z")

</div>

You need to add `processors` directive to `filebeat.yml`, not to `system.yml`.

---

<div class="post-metadata">

**Author:** ![Fpernet](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@Fpernet](https://discuss.elastic.co/u/Fpernet)\
**Post date:** [August 15, 2018, 1:00pm UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/9 "2018-08-15T13:00:25Z")

</div>

Thx Noémi ... so i guess on the top of the file, not in the input section, right ? many thx

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 12, 2018, 1:00pm UTC](https://discuss.elastic.co/t/filtering-processors-do-not-work-for-syslog/144301/10 "2018-09-12T13:00:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.