How i to parse log

[edwinhsieh]

Hi everyone,

I'm a novice to handle log with grok pattern on logstash.

I have a log like that:
[32mINFO \u001b[0m[03-31|09:09:27|print/print.go:20] fire a hole in the room, give you a burst way \tprint=testing tickat=2018-03-31T09:09:27+0000 hash=0x12345 previous=no after=true drop=event:sandbox events=\"bank did added\"

My grok patten filter can be follows:
%{NOTSPACE:notSpace}\[%{DATA:textColor}%{LOGLEVEL:logLevel} %{NOTSPACE:notSpace}\[%{DATA:textColor}\[%{MONTHNUM:month}-%{MONTHDAY:day}\|%{TIME:HHmmss}\|%{GREEDYDATA:logPath}\] %{GREEDYDATA:logContext} %{GREEDYDATA:keyPairs}

But the filter result doesn't follow my request
{
"HHmmss": "07:51:53",
"textColor": "32m",
"logLevel": "INFO",
"month": "03",
"logPath": "print/print.go:20",
"keyPairs": "added"",
"logContext": "fire a hole in the room, give you a burst way print=testing tickat=2018-03-31T07:51:53+0000 hash=0x12345 previous=no after=true drop=event:sandbox events="bank did",
"day": "31",
"notSpace": "\u001b"
}

The keyPairs string should be

print=testing tickat=2018-03-31T07:51:53+0000 hash=0x12345 previous=no after=true drop=event:sandbox events="bank did added"

and logContext is

fire a hole in the room, give you a burst way

Could anyone please give me some advice?

[magnusbaeck (Magnus Bäck)]

What indicates the beginning of the key/value pairs? The string "print="? Or the first "key=value" string? Something else?

For a human it's pretty obvious that the key/value pairs begin with "print=testing" but how can this be formalized?

[edwinhsieh]

"=" indicates key-value pairs while space separates the pairs.

My original problem is how to split a logContext into message and key/value pairs string.

message is mean that

fire a hole in the room, give you a burst way

key/value pairs string is

print=testing tickat=2018-03-31T07:51:53+0000 hash=0x12345 previous=no after=true drop=event:sandbox events="bank did added"

The purpose of key/value pairs string is that i can tell KV filter what's my source.

But I found a magical and powerful KV filter on logstash, it can solve my problem.

I pass the logContext
filter { kv { source => "logContext" } }
and it feedback the KV result.
{ print=testing tickat=2018-03-31T07:51:53+0000 hash=0x12345 previous=no after=true drop=event:sandbox events="bank did added" }

But if i can get the message(fire a hole in the room, give you a burst way) by grok pattern it would be great!

[magnusbaeck (Magnus Bäck)]

Okay, so something like

... %{DATA:logContext} (?<keyPairs>\w+=\S+ .*)

should work, i.e. store everything in logContext until you've found a \w+=\S+ sequence

[edwinhsieh]

thank you!!!