Using ingest pipeline to parse two different log formats with multiple grok pattern


#1

I'm using ingest nodes with pipeline. My log lines basically has two different formats. Basically we want to add another field named "controller" and be able to index it. Sample log lines:

2017-10-04 18:23:42 ERROR [34] GetAsync Exception, Url: api/vehicle/v2/inventory?bac=112093&vin=1GNSCBKC1FR507889&sellingsource=13&bfc=01&sourceZip=&destinationZip=&webId=&type=New&userId=, Response Content:

2017-10-04 18:21:55 ERROR [29] controller:|DS.Marketing.Platform.Deal.V1.Controllers.EligibilityController| DS.Marketing.Platform.Deal.V1.Domain.DeskingProviderNotFoundException: No desking provider has been assigned to this deal (9aac0c8e-a099-449e-9119-c79dbf937abc)

Here is my ingest pipeline but it's far from complete. Please help!

POST _ingest/pipeline/_simulate
{
"pipeline" :
{
"description": "grok_controller_mp_combined",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{CUSTOM_CONTROLLER:log.controller}"],
"pattern_definitions" : {
"CUSTOM_CONTROLLER (^\s*controller:.+)"
}
}
}]},

filter{
if ""%{CUSTOM_CONTROLLER:log.controller}" in [message]
{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.controller}" } }
}
else
{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.message}" } }
}
}
"docs":[
{
"_source": {"message":
"2017-05-09 14:51:37 ERROR [95] System.ArgumentException: Lead.PersonalIdentificationNumber or BusinessTaxId are required. \n at MP.StarbodLeadProcessing.Processing.StarbodMapping.AR.StarbodMappingSiebelArgentina.Validate(StarbodModel starbodModel"}
},
{
"_source": {"message":
"2017-10-04 18:23:42 ERROR [34] GetAsync Exception, Url: api/vehicle/v2/inventory?bac=112093&vin=1GNSCBKC1FR507889&sellingsource=13&bfc=01&sourceZip=&destinationZip=&webId=&type=New&userId=, Response Content: "}
},
{
"_source" {"message":"2017-10-04 18:21:55 ERROR [29] controller:|DS.Marketing.Platform.Deal.V1.Controllers.EligibilityController| DS.Marketing.Platform.Deal.V1.Domain.DeskingProviderNotFoundException: No desking provider has been assigned to this deal (9aac0c8e-a099-449e-9119-c79dbf937abc)."}
]
}


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.