Grok pattern works but not applied on Kibana (2nd)

[Cuju]

Hello, I have an issue I try to fix for a couple of days, without success. I posted it on the wrong channel so here we go again.

I need to grok the message field (they don't have the same syntax) from my Kibana but it fails.

I created my grok filter, which works on Grok Debugger :
%{HOSTNAME:hote}_%{DATA:val} %{WORD:process}\\[%{NUMBER:procces_pid}\\]: %{DATA:msg}: %{WORD:protocole} %{WORD:peer} %{IP:client} \\(%{DATA:AS}%{NUMBER:AS_nb}\\) %{DATA:etat} \\(%{DATA:evenement}\\) \\(%{DATA:instance}\\)
for this type of message :
ig1-edge-dc3-01_re0 rpd[1524]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 158.58.176.35 (External AS 200271) changed state from EstabSync to Established (event RsyncAck) (instance master) . I created a pipeline for the grok, without any error but it doesn't parse the message.

At first, I tried to create a pipeline :

PUT /_ingest/pipeline/filetest
{
  "description": "Pipeline for parsing Syslog messages.",
  "processors": [
    {
      "grok": {
        "ignore_missing": true,
        "field": "message",
        "patterns": [
          "%{HOSTNAME:hote}_%{DATA:val} %{WORD:process}\\[%{NUMBER:procces_pid}\\]: %{DATA:msg}: %{WORD:protocole} %{WORD:peer} %{IP:client} \\(%{DATA:AS}%{NUMBER:AS_nb}\\) %{DATA:etat} \\(%{DATA:evenement}\\) \\(%{DATA:instance}\\)"
        ]
      }
    }
  ]
}

but it didn't work, so I tried to add my grok to one of the preexistent pipelines (it doesn't have my grok in it) :

PUT /_ingest/pipeline/filebeat-7.5.2-system-syslog-pipeline
    {
                "description" : "Pipeline for parsing Syslog messages.",
                "processors" : [
                  {
                    "grok" : {
                      "pattern_definitions" : {
                        "GREEDYMULTILINE" : "(.|)*"
                      },
                      "ignore_missing" : true,
                      "field" : "message",
                      "patterns" : [
                        "%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name} (?:\\[%{POSINT:process.pid:long}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
                        "%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}",
                        "%{TIMESTAMP_ISO8601:system.syslog.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{GREEDYMULTILINE:system.syslog.message}"
                      ]
                    }
                  },
                  {
                    "rename" : {
                      "field" : "system.syslog.message",
                      "target_field" : "message",
                      "ignore_missing" : true
                    }
                  },
                  {
                    "date" : {
                      "formats" : [
                        "MMM  d HH:mm:ss",
                        "MMM dd HH:mm:ss",
                        "MMM d HH:mm:ss",
                        "ISO8601"
                      ],
                      "on_failure" : [
                        {
                          "append" : {
                            "field" : "error.message",
                            "value" : "{{ _ingest.on_failure_message }}"
                          }
                        }
                      ],
                      "if" : "ctx.event.timezone == null",
                      "field" : "system.syslog.timestamp",
                      "target_field" : "@timestamp"
                    }
                  },
                  {
                    "date" : {
                      "if" : "ctx.event.timezone != null",
                      "field" : "system.syslog.timestamp",
                      "target_field" : "@timestamp",
                      "formats" : [
                        "MMM  d HH:mm:ss",
                        "MMM dd HH:mm:ss",
                        "MMM d HH:mm:ss",
                        "ISO8601"
                      ],
                      "timezone" : "{{ event.timezone }}",
                      "on_failure" : [
                        {
                          "append" : {
                            "field" : "error.message",
                            "value" : "{{ _ingest.on_failure_message }}"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "remove" : {
                      "field" : "system.syslog.timestamp"
                    }
                  }
                ],
                "on_failure" : [
                  {
                    "set" : {
                      "field" : "error.message",
                      "value" : "{{ _ingest.on_failure_message }}"
                    }
                  }
                ]
              }

and it also fails pitifully.

Do you have any idea ? Thanks a lot.

[Cuju]

I finally fix it.

Actually, I update FIlebeat but it was still looking at the 7.5 version instead of the 7.6. A directory for the 7.6 version was created, but not on the /etc/filebeat/ so I had to move the files to take over the old one.

Furthermore, I had issues that I was able to fix thanks to the simulation.