Logstash/Grok help

[mduckett1 (Mike Duckett)]

Hi,

I am trying to pull in some logs using filebeat and having some trouble (mainly because of lack of understanding I think).

The log file has multiple lines like this:

2017-04-09 12:53:10 INFO: InsertedAt=2017-04-09 11:52:34; EventID=XXXX; EventTime=2017-04-09 11:52:33; EventTypeID=X; EventType=XXXXX; Name=; ReportingName=XXXXX; UserName=XXXXX; ActionID=X; Action=XXXX; SubTypeID=; SubType=; DeviceTypeID=x; DeviceType=XXXXX; Model=XXXXX; DeviceID=; ComputerName=XXXXX;

I'd like to extract each item into its own field, so started with a grok filter to get the timestamp and the rest of the message:

["message", "%{DATESTAMP:timestamp} INFO: %{GREEDYDATA:filebeat_message"]

This works on a grok test site, but it is being tagged with _grokparsefailure by logstash.

I'd like to then split the rest of the message into the fields seperated by ;

Hope that makes sense, sorry Im quite new to this and learning as I go!

Thanks,

Mike

[magnusbaeck (Magnus Bäck)]

DATESTAMP won't work but TIMESTAMP_ISO8601 should work. When you've got the grok filter working, feed filebeat_message to a kv filter.

[mduckett1 (Mike Duckett)]

Thanks Magnus, I had just started looking at kv filter and using this:

kv {
source => "filebeat_message"
value_split => "="
field_split => "; "
}

Seems to split it up as required, the grok filter doesnt seem to work though even changing to TIMESTAMP_ISO8601 I still get a _grokparsefailure.

However Im guessing I should be able to just do this with the kv filter shouldnt i? Once its been through that I can use a date to set the timestamp to EventTIme?

[magnusbaeck (Magnus Bäck)]

• I'm quite sure TIMESTAMP_ISO8601 works.
• Don't include a space in the field_split value. That value is not a multi-character string but a character class, so if you include a space then spaces will also be field splitters. Use the trim option to remove the extra spaces. Check the kv filter docs for your version of Logstash for the exact option name.
• Yes, you should probably use the EventTime value instead, but you'll still want to use grok to extract the key/value pairs form the string. Otherwise the first first is going to be "2017-04-09 12:53:10 INFO: InsertedAt".

[mduckett1 (Mike Duckett)]

Thanks Magnus, in the end it turned out there was another config file being used that had no if statement on it's filter causing this _grokparsefailure. All seems to be working correctly now.