I'm using ingest nodes with pipeline. My log lines basically has two different formats. Basically we want to add another field named "controller" and be able to index it. Sample log lines:
2017-10-04 18:23:42 ERROR [34] GetAsync Exception, Url: api/vehicle/v2/inventory?bac=112093&vin=1GNSCBKC1FR507889&sellingsource=13&bfc=01&sourceZip=&destinationZip=&webId=&type=New&userId=, Response Content:
2017-10-04 18:21:55 ERROR [29] controller:|DS.Marketing.Platform.Deal.V1.Controllers.EligibilityController| DS.Marketing.Platform.Deal.V1.Domain.DeskingProviderNotFoundException: No desking provider has been assigned to this deal (9aac0c8e-a099-449e-9119-c79dbf937abc)
Here is my ingest pipeline but it's far from complete. Please help!
POST _ingest/pipeline/_simulate
{
"pipeline" :
{
"description": "grok_controller_mp_combined",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{CUSTOM_CONTROLLER:log.controller}"],
"pattern_definitions" : {
"CUSTOM_CONTROLLER (^\s*controller:.+)"
}
}
}]},
filter{
if ""%{CUSTOM_CONTROLLER:log.controller}" in [message]
{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.controller}" } }
}
else
{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.message}" } }
}
}
"docs":[
{
"_source": {"message":
"2017-05-09 14:51:37 ERROR [95] System.ArgumentException: Lead.PersonalIdentificationNumber or BusinessTaxId are required. \n at MP.StarbodLeadProcessing.Processing.StarbodMapping.AR.StarbodMappingSiebelArgentina.Validate(StarbodModel starbodModel"}
},
{
"_source": {"message":
"2017-10-04 18:23:42 ERROR [34] GetAsync Exception, Url: api/vehicle/v2/inventory?bac=112093&vin=1GNSCBKC1FR507889&sellingsource=13&bfc=01&sourceZip=&destinationZip=&webId=&type=New&userId=, Response Content: "}
},
{
"_source" {"message":"2017-10-04 18:21:55 ERROR [29] controller:|DS.Marketing.Platform.Deal.V1.Controllers.EligibilityController| DS.Marketing.Platform.Deal.V1.Domain.DeskingProviderNotFoundException: No desking provider has been assigned to this deal (9aac0c8e-a099-449e-9119-c79dbf937abc)."}
]
}