Parsing a log with colon separated fields

[verno_s (Scott Vernon)]

Hi,

I've tried to read the Logstash docs on grok parsing but i can't find an answer to my issue.

I want to parse a : separated logfile e.g.

29/11/2016 11:49:17 AM : Response: 125 Data connection already open; Transfer starting.
29/11/2016 11:54:18 AM : Response: 226 Transfer complete.
29/11/2016 11:54:18 AM : Command : PWD

If i use the following match statement the hours and minutes from the timestamp are mistakenly parsed as fields.

:%{DATA:log_timestamp}:%{DATA:action}:%{GREEDYDATA:message}

Output from grokconstructor:

MATCHED
log_timestamp 49
action 17·AM·
message ·Response:·125·Data·connection·already·open;·Transfer·starting.
before match: 29/11/2016 11
29/11/2016 11:54:18 AM : Response: 226 Transfer complete.
MATCHED
log_timestamp 54
action 18·AM·
message ·Response:·226·Transfer·complete.
before match: 29/11/2016 11
29/11/2016 11:54:18 AM : Command : PWD
MATCHED
log_timestamp 54
action 18·AM·
message ·Command·:·PWD
before match: 29/11/2016 11

I would like the the following result, what is the best way to do this, is it to create a custom pattern?

log_timestamp 29/11/2016 11:49:17 AM
action Response
message 125 Data connection already open; Transfer starting.

Many thanks,

Scott

[magnusbaeck (Magnus Bäck)]

Using more than one DATA or GREEDYDATA in the same expression is asking for trouble. Use more specific patterns for matching the timestamp. The grok constructor web site should be able to help with suggestions. For example, you can use %{DATE_EU} %{TIME} to match the timestamp (except AM/PM).

[Christian_Dahlqvist]

Based on the log entries you presented, it looks like the fields are actually separated by' : ' and not just a':'. If you add these spaces I suspect your expression should work. Finding more specific patterns than DATA would however as Magnus suggests improve parsing.

[verno_s (Scott Vernon)]

Thanks guys! Much appreciated. I have modified the delimiter to be ' : ' (with spaces) I've also changed one of the DATA's to a WORD and will investigate using more specific patterns to match the date.

Thanks again for the quick suggestions.

Scott