Match and extract last part of log in pipe delimited log

Elastic Stack Logstash
1

Logstash is receving following log -
> Timestamp|Field1|Field2|...|Field n|Log message

The number of fields is varaible and not fixed.
My goal is to extract just the Timestamp and Log message and add it as a new key value pair in the document as shown below-

{
"Timestamp": "2017-06-15T09:39:02.619Z",
"message": "Log message"
}

I tried using grok filter. I can get the timestamp using its 'match' option.

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:Timestamp}" }
}

But I am unable to match the log message as the fields in between are variable.

Is it possible to do this using grok filter or can any other filter help?

2

Grok should be fine.
After the %{TIMESTAMP_ISO8601:Timestamp}\| allow for the first pipe, then allow for a variable number of ([^|]+\|)+ then the message %{GREEDYDATA:message}

See http://grokconstructor.appspot.com/do/match
I used this Grok pattern

%{TIMESTAMP_ISO8601:Timestamp}\|(?:[^|]+\|)+%{GREEDYDATA:msg}

I used these lines

2017-06-15T09:39:02.619Z|A|B|C|D|E|this is a long message
2017-06-15T09:39:04.619Z|A|B|E|this is a second long message

I got this result

2017-06-15T09:39:02.619Z|A|B|C|D|E|this is a long message
MATCHED
Timestamp 2017-06-15T09:39:02.619Z
msg       this·is·a·long·message
2017-06-15T09:39:04.619Z|A|B|E|this is a second long message
MATCHED
Timestamp 2017-06-15T09:39:04.619Z
msg       this·is·a·second·long·message
1 Like
3

Thanks a lot Guy. It helped me.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.