Goal is: for each line in the log, there should be a document in elastic containing the 'message' (text after time stamp). Each document should also contain fields for the the project name, plan name, and build #. <--this is where I'm getting stuck
example log snippet (first two lines, labeled line 1 and 2):
line 1: simple 01-Jan-2016 14:26:01 Build TestProj - Framework Code - Build #25 (TST-FC-25) started building on agent .NET Core 2
line 2: simple 01-Jan-2016 14:26:01 .NET-related builds, tests and publishing.
I have a Grok to get and create the fields I want - build name, build number, and project name (and have them as fields in elastic):
%{NOTSPACE:log_entrytype}%{SPACE}(?<timestamp>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])-\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b-(?>\d\d){1,2}\s*(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))%{SPACE}Build%{SPACE}%{DATA:BamProjName}%{SPACE}-%{SPACE}%{DATA:BamBuildName}%{SPACE}-%{SPACE}Build%{SPACE}#%{NUMBER:BamBuildNum}
Is my end goal achievable with logstash and the Grok plugin alone? Or do I need ruby here ?
***NOTE: I'm using filebeat for shipping logs, and elastic does not recommend the multiline codec , so I'm curious what my other options are (assuming the multiline codec is even a viable option).
Going the ruby route assuming that is the best way...here is what I have which creates the field in elastic however it is coming up as "-" (I assume that means null or empty):
code => "
@@projName = 'lineBelowSeemsToPopulateWithEmptyStr'
@@projName = event['BamProjName']
event['BamProjectName'] = @@projName if
@@projName.nil? || @@projName.empty?
"