Logstash grok parse error parsing log file

stefansaye · May 3, 2016, 6:50am

Hello,
i try to parse my log and get these values looks like

interfaces => PP/0/P1/CPU0
timestamp => 2016-05-03 19:37:08.879
message => mp[1051]: %ROUTING-MP-5-INIT_PEER_UP_DOWN : MP peer down: 1.2.3.4 : received socket disconnect notification

syslog input, as follows:

"message" => "<189>26739314: PP/0/P1/CPU0:May 3 19:37:08.879 : mp[1051]: %ROUTING-MP-5-INIT_PEER_UP_DOWN : MP peer down: 1.2.3.4 : received socket disconnect notification \n",

with this config file:

filter
 { 
     grok {
    match => {"message" => "{<%{POSINT:syslog_pri}>%{NUMBER:id}:%{SYSLOGBASE:interface}:% 
    {SYSLOGTIMESTAMP:syslog_timestamp}:%{GREEDYDATA:syslog_message}"} }
 }

However, I am getting a grok prase failure, I am not sure what the problem is. cant seem to pin point the pattern that is causing the problem. Any thoughts/comments would be appreciated.

bhatch · May 3, 2016, 5:32pm

Have you tried using http://grokconstructor.appspot.com/do/match

If I copy paste your message field and grok patter into the window I get this result. You can see that the first two patterns matched, but nothing after that did. It looks to be breaking around the SYSLOGBASE tag.

This link has all of the GROK expression definitions.

github.com

logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b

POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
# URN, allowing use of RFC 2141 section 2.3 reserved characters

This file has been truncated. show original

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

SYSLOGBASE looks to start by looking for some kind of timestamp. The remaining text doesn't begin with that timestamp so it fails.

stefansaye · May 4, 2016, 2:54am

i have modify my pattern as below,

 grok {
  match => {"message" => "<%{POSINT:syslog_pri}>%{NUMBER:syslog_pid}: %{DATA:interface}:%{SYSLOGTIMESTAMP:syslog_timestamp}:%{GREEDYDATA:syslog_message}"}
   }

it's work in http://grokconstructor.appspot.com/do/match
but get "[0] "_grokparsefailure"" when run logstash

Can you please tell me what happened ?
Any thoughts/comments would be appreciated.

bhatch · May 4, 2016, 5:35pm

It actually still showed as not fully matching for me.

If you look at your test line you can see that it has a space right after the timestamp, I don't see that on yours.
Add a space between the colon separating the two expressions.
Before:
%{SYSLOGTIMESTAMP:syslog_timestamp}:%{GREEDYDATA:syslog_message}
After:
%{SYSLOGTIMESTAMP:syslog_timestamp} : %{GREEDYDATA:syslog_message}

Once I made that change I now see: