Match and extract last part of log in pipe delimited log

Logstash is receving following log -
> Timestamp|Field1|Field2|...|Field n|Log message

The number of fields is varaible and not fixed.
My goal is to extract just the Timestamp and Log message and add it as a new key value pair in the document as shown below-

{
"Timestamp": "2017-06-15T09:39:02.619Z",
"message": "Log message"
}

I tried using grok filter. I can get the timestamp using its 'match' option.

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:Timestamp}" }
}

But I am unable to match the log message as the fields in between are variable.

Is it possible to do this using grok filter or can any other filter help?

Grok should be fine.
After the %{TIMESTAMP_ISO8601:Timestamp}\| allow for the first pipe, then allow for a variable number of ([^|]+\|)+ then the message %{GREEDYDATA:message}

See http://grokconstructor.appspot.com/do/match
I used this Grok pattern

%{TIMESTAMP_ISO8601:Timestamp}\|(?:[^|]+\|)+%{GREEDYDATA:msg}

I used these lines

2017-06-15T09:39:02.619Z|A|B|C|D|E|this is a long message
2017-06-15T09:39:04.619Z|A|B|E|this is a second long message

I got this result

2017-06-15T09:39:02.619Z|A|B|C|D|E|this is a long message
MATCHED
Timestamp 2017-06-15T09:39:02.619Z
msg       thisĀ·isĀ·aĀ·longĀ·message
2017-06-15T09:39:04.619Z|A|B|E|this is a second long message
MATCHED
Timestamp 2017-06-15T09:39:04.619Z
msg       thisĀ·isĀ·aĀ·secondĀ·longĀ·message
1 Like

Thanks a lot Guy. It helped me.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.