Change Grok in Filebeat (Log Input)


I've found the file /usr/share/filebeat/module/system/syslog/ingest/pipeline.json which parses syslog messages.

but it parses just the format of Redhat Ubuntu and so on.

Format of RHEL:
Aug 14 14:20:48 server1 systemd: Started Fingerprint Authentication Daemon.
Aug 14 14:20:48 server1 fprintd: Launching FprintObject

now i have suse with a different time format.
I wanted to edit the grok filter in the module but always fails.

Suse Format:

2018-08-14T14:30:02.203151+02:00 linux-sqrz systemd[4179]: Stopped target Basic System.
2018-08-14T14:30:02.203251+02:00 linux-sqrz systemd[4179]: Stopped target Paths.

I alway get the message "Provided Grok expressions do not match field value: ........" in Kibana

At the end, i just want to have the same format for Suse or Redhat Logs in Kibana

Here is my config File:

        "description": "Pipeline for parsing Syslog messages.",
        "processors": [
                        "grok": {
                                "field": "message",
                                "patterns": [
                                        "%{TIMESTAMP_ISO8601:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
                                        "%{TIMESTAMP_ISO8601:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"
        "pattern_definitions" : {
          "GREEDYMULTILINE" : "(.|\n)*"
        "ignore_missing": true
      "remove": {
        "field": "message"
      "date": {
        "field": "system.syslog.timestamp",
        "target_field": "@timestamp",
        "formats": [
                                        "MMM  d HH:mm:ss",
                                        "MMM dd HH:mm:ss",
                                        "YYYY MM dd HH:mm:ss"
        "timezone": "Europe/Amsterdam",
        "ignore_failure": true
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"

I really don't know changing the ppeline.json can ingest the custom logs from filebeat modules but in your config i can see the date format as "YYYY MM dd HH:mm:ss" but this should be "YYYY-MM-dd HH:mm:ss.SSSSSS" with the format of the Suse which you mentioned on top.

I removed the date filter. And it still shows the same message. but it still shows the same message. I also tried the grok debugger in kibana and there it works.
I dont know where the problem is

Are you sure that the correct version of the pipeline is loaded into ES? By default pipelines are not updated if Filebeat tries to load a new pipeline with a same ID.

To force loading the new pipeline you could run the following: ./filebeat setup --pipelines -modules="system".

Thank you very much. it works now

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.