Grok pattern not getting full message

Hello!

I have have a problem with grok getting full message from log file. Can you please help me?

Log message:

2019-10-29 19:27:21.779+02:00 [ 27] INFO - State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal
Changes based on data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2116,678,

Grok pattern:

%{TIMESTAMP_ISO8601:system.syslog.timestamp} [ %{SPACE} %
{NUMBER:milestone_lognum}] %{LOGLEVEL:milestone_loglevel} %{SPACE} - %{GREEDYDATA:milestone_message}

Structured data result:

{
"system": {
"syslog": {
"timestamp": "2019-10-29 19:27:21.779+02:00"
}
},
"milestone_lognum": "27",
"milestone_loglevel": "INFO",
"milestone_message": "State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal "
}

Could it be because you return to ligne after Normal ?

Try running your Grok pattern with this log

2019-10-29 19:27:21.779+02:00 [ 27] INFO - State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal Changes based on data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2116,678,

Exactly the message after Normal is in new line in the source log file, but i need to parse this as one whole message field.

Have you tried setting multiline.pattern? see more:
https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

I did grok debugging on this log:

2019-10-31 09:13:10.178+02:00 [ 20] INFO - State has changed: Rule: Recording FPS, > Counter: \VideoOS Recording Server Device(c7 [3793243f-0055-4250-b98f-b15792b055e0])\Media/sec perc, Before: Warning, now: Normal
Changes based on data: 208,0519, 0, 0, 0, 208,4882, 0, 0, 415,8712, 0, 0, 0, 0, 208,2306, 0, 0, 0, 208,4588, 0, 0, 0, 208,1291, 0, 0, 0, 208,338, 0, 0, 0, 207,3737,

With this grok pattern:

%{TIMESTAMP_ISO8601:system.syslog.timestamp} \[ %{SPACE} %{NUMBER:milestone_lognum}\] %{LOGLEVEL:milestone_loglevel} %{SPACE} \- (?<milestone_message>(.|\r|\n)*)

Structured data:

{
"system": {
"syslog": {
"timestamp": "2019-10-31 09:13:10.178+02:00"
}
},
"milestone_lognum": "20",
"milestone_loglevel": "INFO",
"milestone_message": "State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(c7 [3793243f-0055-4250-b98f-b15792b055e0])\Media/sec perc, Before: Warning, now: Normal \n Changes based on data: 208,0519, 0, 0, 0, 208,4882, 0, 0, 415,8712, 0, 0, 0, 0, 208,2306, 0, 0, 0, 208,4588, 0, 0, 0, 208,1291, 0, 0, 0, 208,338, 0, 0, 0, 207,3737,"
}

But in elasticsearch i still get grok parse failure events for new line :frowning:

Hi @tomsozolins

I used the Heroku Grok Debugger to analyze your pattern. The following pattern returned matches:

%{TIMESTAMP_ISO8601:system.syslog.timestamp} [ %{NUMBER:milestone_lognum}] %{LOGLEVEL:milestone_loglevel} - (?<milestone_message>(.|\r|\n)*)

I think the mistake in your pattern were those %{SPACE}.

Hope this helps.

Tried without spaces. It matches in heroku website, but does not match in elasticsearch grok debugger. There is a space at the end of the first line and also at the start of the second line.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.