Change Grok in Filebeat (Log Input)

Elastic Stack Beats
1

Hello

I've found the file /usr/share/filebeat/module/system/syslog/ingest/pipeline.json which parses syslog messages.

but it parses just the format of Redhat Ubuntu and so on.

Format of RHEL:
Aug 14 14:20:48 server1 systemd: Started Fingerprint Authentication Daemon.
Aug 14 14:20:48 server1 fprintd: Launching FprintObject

now i have suse with a different time format.
I wanted to edit the grok filter in the module but always fails.

Suse Format:

2018-08-14T14:30:02.203151+02:00 linux-sqrz systemd[4179]: Stopped target Basic System.
2018-08-14T14:30:02.203251+02:00 linux-sqrz systemd[4179]: Stopped target Paths.

I alway get the message "Provided Grok expressions do not match field value: ........" in Kibana

At the end, i just want to have the same format for Suse or Redhat Logs in Kibana

Here is my config File:

{
        "description": "Pipeline for parsing Syslog messages.",
        "processors": [
                {
                        "grok": {
                                "field": "message",
                                "patterns": [
                                        "%{TIMESTAMP_ISO8601:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
                                        "%{TIMESTAMP_ISO8601:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"
                                ],
        "pattern_definitions" : {
          "GREEDYMULTILINE" : "(.|\n)*"
        },
        "ignore_missing": true
                        }
    },
                {
      "remove": {
        "field": "message"
      }
    },
    {
      "date": {
        "field": "system.syslog.timestamp",
        "target_field": "@timestamp",
        "formats": [
                                        "MMM  d HH:mm:ss",
                                        "MMM dd HH:mm:ss",
                                        "YYYY MM dd HH:mm:ss"
        ],
        "timezone": "Europe/Amsterdam",
        "ignore_failure": true
      }
    }
        ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}
2

I really don't know changing the ppeline.json can ingest the custom logs from filebeat modules but in your config i can see the date format as "YYYY MM dd HH:mm:ss" but this should be "YYYY-MM-dd HH:mm:ss.SSSSSS" with the format of the Suse which you mentioned on top.

3

I removed the date filter. And it still shows the same message. but it still shows the same message. I also tried the grok debugger in kibana and there it works.
I dont know where the problem is

4

Are you sure that the correct version of the pipeline is loaded into ES? By default pipelines are not updated if Filebeat tries to load a new pipeline with a same ID.

To force loading the new pipeline you could run the following: ./filebeat setup --pipelines -modules="system".

5

Thank you very much. it works now

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.