Removing part of message giving grokparsefailure

[jlim0930 (J)]

while sending logs from a central syslog server it adds "Message forwarded from ...." in the message field.

I added
if "Message forwarded from" in [message] {
mutate {
gsub => [
"message", "Message forwarded from ", "",
"message", "<.*>", ""
]
}
}
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
}
}

now the "Message forwarded from" is gone from the message but now i am getting grokparsefailure

[Badger]

What does [message] look like?

[jlim0930 (J)]

just a normal syslog from systems but coming from a centralized syslog server so "message forwarded from" gets added and thats what I am trying to remove

Sep 19 10:46:07 Message forwarded from hostname: program: logs logs losg

i see where the grokparsefailure is coming from... I used mutate to remove "Message forwarded from" but how do you remove the : after the hostname?

[Badger]

Wouldn't it be easier to add the colon to the grok pattern?

[jlim0930 (J)]

is there a way to do grok where the first : can exist or not and still parse ok?

like
Sep 19 10:46:07 Message forwarded from hostname: program: logs logs losg
vs
Sep 19 10:46:07 Message forwarded from hostname program: logs logs losg

[Badger]

()? is used for optional fields, as you have used for the syslog_pid, so

"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname}(:)? %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}"

[jlim0930 (J)]

Great! working like a champ! thanks