Hi,
I am trying to implement filebeat multiline pattern on our custom application logs which contains entries of different formats as shown below.
[2019-09-27T10:35:44.257+02:00] [osb_server1] [NOTIFICATION] [oracle.osb.logging.pipeline] [tid: 65] [userId: ] [ecid: db6c52be-d674-418a-bb79-b6e5aa5ac6a8-00075673,0] [APP: Service Bus Logging] [FlowId: 0000Mplr9363JBl6wvvX6G1TZDkH0006UW] [GetMessageFromTakeCargoQueue, GetMessageFromTakeCargoQueue_request, LoopForRegisteredAskoCompainesInTakeCargo, REQUEST] <<<<<< No of deleted Messages >>>>>>: 0
[2019-09-27T10:35:44.262+02:00] [osb_server1] [NOTIFICATION] [oracle.osb.logging.pipeline] [tid: 65] [userId: ] [ecid: db6c52be-d674-418a-bb79-b6e5aa5ac6a8-00075673,0] [APP: Service Bus Logging] [FlowId: 0000Mplr9363JBl6wvvX6G1TZDkH0006UW] [CSF_TrackMessage_To_Elastic, request-N53ed173e.N32fad4e5.0.16871088aa4.N7fc4, Log, REQUEST] <ns0:trackMessage xmlns:WL5G3N1="urn:company.no/Services/AnyXMLService" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns0="http://xmlns.oracle.com/TrackingDetails">[[
ns0:header
ns0:isDetailsRequired
ns0:isDetailsRequiredY</ns0:isDetailsRequired>
ns0:queryDetailsBy1</ns0:queryDetailsBy>
</ns0:isDetailsRequired>
</ns0:header>
</ns0:trackMessage>
]]
[2019-09-27T11:03:44.608+02:00] [osb_server1] [NOTIFICATION] [oracle.osb.logging.pipeline] [tid: 106] [userId: ] [ecid: db6c52be-d674-418a-bb79-b6e5aa5ac6a8-00077478,0] [APP: Service Bus Logging] [FlowId: 0000MplxZRu3JBl6wvvX6G1TZDkH0006ZV] [GetMessageFromTakeCargoQueue, GetMessageFromTakeCargoQueue_request, LoopForRegisteredAskoCompainesInTakeCargo, REQUEST] <<<<<< No of deleted Messages >>>>>>: 0
[2019-09-27T11:03:44.671+02:00] [osb_server1] [NOTIFICATION] [oracle.osb.logging.pipeline] [tid: 106] [userId: ] [ecid: db6c52be-d674-418a-bb79-b6e5aa5ac6a8-00077478,0] [APP: Service Bus Logging] [FlowId: 0000MplxZRu3JBl6wvvX6G1TZDkH0006ZV] [GetMessageFromTakeCargoQueue, GetMessageFromTakeCargoQueue_request, LoopForRegisteredAskoCompainesInTakeCargo, REQUEST] <<<<<< No of deleted Messages >>>>>>: 0
[2019-09-27T11:03:44.916+02:00] [osb_server1] [NOTIFICATION] [oracle.osb.logging.pipeline] [tid: 106] [userId: ] [ecid: db6c52be-d674-418a-bb79-b6e5aa5ac6a8-00077478,0] [APP: Service Bus Logging] [FlowId: 0000MplxZRu3JBl6wvvX6G1TZDkH0006ZV] [CSF_TrackMessage_To_Elastic, request-N53ed173e.N32fad4e5.0.16871088aa4.N7fc4, Log, REQUEST] <ns0:trackMessage xmlns:WL5G3N1="urn:company.no/Services/AnyXMLService" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns0="http://xmlns.oracle.com/TrackingDetails">[[
ns0:header
ns0:isDetailsRequired
ns0:isDetailsRequiredY</ns0:isDetailsRequired>
ns0:queryDetailsBy1</ns0:queryDetailsBy>
</ns0:isDetailsRequired>
</ns0:header>
</ns0:trackMessage>
]]
I want filbeat read only lines which start with timestamp,contains "To_Elastic" and append all the following lines until it encounters "]]".
I am trying with the below multiline configuration but this also picks all those lines which doesn't contain "To_Elastic" and clubs them into a single event.
multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}.To_Elastic.'
multiline.max_lines: 50000
multiline.flush_pattern: '^]]'
multiline.timeout: 100
multiline.negate: true
multiline.match: after
Requesting help in fixing this. Thanks in advance.