Grok pattern not getting full message

Elastic Stack Elasticsearch
1

Hello!

I have have a problem with grok getting full message from log file. Can you please help me?

Log message:

2019-10-29 19:27:21.779+02:00 [ 27] INFO - State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal
Changes based on data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2116,678,

Grok pattern:

%{TIMESTAMP_ISO8601:system.syslog.timestamp} [ %{SPACE} %
{NUMBER:milestone_lognum}] %{LOGLEVEL:milestone_loglevel} %{SPACE} - %{GREEDYDATA:milestone_message}

Structured data result:

{
"system": {
"syslog": {
"timestamp": "2019-10-29 19:27:21.779+02:00"
}
},
"milestone_lognum": "27",
"milestone_loglevel": "INFO",
"milestone_message": "State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal "
}

2

Could it be because you return to ligne after Normal ?

Try running your Grok pattern with this log

2019-10-29 19:27:21.779+02:00 [ 27] INFO - State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal Changes based on data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2116,678,
3

Exactly the message after Normal is in new line in the source log file, but i need to parse this as one whole message field.

4

Have you tried setting multiline.pattern? see more:
https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

6

I did grok debugging on this log:

2019-10-31 09:13:10.178+02:00 [ 20] INFO - State has changed: Rule: Recording FPS, > Counter: \VideoOS Recording Server Device(c7 [3793243f-0055-4250-b98f-b15792b055e0])\Media/sec perc, Before: Warning, now: Normal
Changes based on data: 208,0519, 0, 0, 0, 208,4882, 0, 0, 415,8712, 0, 0, 0, 0, 208,2306, 0, 0, 0, 208,4588, 0, 0, 0, 208,1291, 0, 0, 0, 208,338, 0, 0, 0, 207,3737,

With this grok pattern:

%{TIMESTAMP_ISO8601:system.syslog.timestamp} \[ %{SPACE} %{NUMBER:milestone_lognum}\] %{LOGLEVEL:milestone_loglevel} %{SPACE} \- (?<milestone_message>(.|\r|\n)*)

Structured data:

{
"system": {
"syslog": {
"timestamp": "2019-10-31 09:13:10.178+02:00"
}
},
"milestone_lognum": "20",
"milestone_loglevel": "INFO",
"milestone_message": "State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(c7 [3793243f-0055-4250-b98f-b15792b055e0])\Media/sec perc, Before: Warning, now: Normal \n Changes based on data: 208,0519, 0, 0, 0, 208,4882, 0, 0, 415,8712, 0, 0, 0, 0, 208,2306, 0, 0, 0, 208,4588, 0, 0, 0, 208,1291, 0, 0, 0, 208,338, 0, 0, 0, 207,3737,"
}

But in elasticsearch i still get grok parse failure events for new line :frowning:

7

Hi @tomsozolins

I used the Heroku Grok Debugger to analyze your pattern. The following pattern returned matches:

%{TIMESTAMP_ISO8601:system.syslog.timestamp} [ %{NUMBER:milestone_lognum}] %{LOGLEVEL:milestone_loglevel} - (?<milestone_message>(.|\r|\n)*)

I think the mistake in your pattern were those %{SPACE}.

Hope this helps.

8

Tried without spaces. It matches in heroku website, but does not match in elasticsearch grok debugger. There is a space at the end of the first line and also at the start of the second line.