Each entry has multiple lines with keywords

[fribse (Kenneth Fribert)]

Hi Guys and Gals

I'm hoping that somebody has run into something similar, and can give me a hint.
I have a lot of logfiles where each key is on a seperate line, and each event is seperated by a row of dashes, like this:

Exception: PD-10000: Error opening provider 'MHTestMessageProvider': PD-10000: Catastrophic failure
ErrorCode: 10000
Time: 22-05-2018 10:28:30
Module: C:\Program Files (x86)\PST\Million Handshakes\Dialog Server\MHDialogServer.dll
Version: 6.1.4.931
File date: 02-02-2018 10:58:02
Instance name: XXXXX
Username: internal
ProcessID: 2088
ThreadID: 156
Memory usage: 121,41 MiB
Exception type: EMHProviderOpenException
Procedure: MHProviderAccessLayer.TMHProviderAccessLayer.GetClientDataset

[07492060] MHProviderAccessLayer.TMHProviderAccessLayer.GetClientDataset
[06C75B47] MHMessageAccessLayer.TMHMessageAccessLayer.DoUpdateTemplateThumbnail
[06C74FD0] MHMessageAccessLayer.TMHMessageAccessLayer.UpdateTemplateThumbnail
[066D1786] System.Win.ComObj.TAutoObject.Invoke
[069BB1EA] MHComUtils.CallMethod2
[069BB476] MHComUtils.CallMethod2
[069BB3DE] MHComUtils.CallMethod2
[06B0A739] MHSystemTaskAccessLayer.TMHSystemTaskAccessLayer.ProcessSystemTask
[075F9BEC] MHSystemTaskExecutionCommand.TMHSystemTaskExecutionCommand.InternalExecute
[075FDE94] MHInternalServiceSupport.TMHInternalServiceSupport.ProcessSystemTask
--------------------------------------------------------------------------------

So far I've managed to get the multiline to grab the entire last section as one event, but the problem is that each line is treated as seperate events by adding this:

  multiline.pattern: ^\[(\S{8})\]
  multiline.negate: false
  multiline.match: after

I would very much like to group them as one event.

I hope somebody can understand this fumbling start on filebeats, and help me out

Best regards
Kenneth

[adrisr (Adrian Serrano)]

Try with:

multiline.pattern: '^--*$'
multiline.negate: true
multiline.match: before

[fribse (Kenneth Fribert)]

Hi Adrian

Thankyou for your answer, it is much appreciated.
Wouldn't I loose the 'keys' for each line with this, and just get a big lump of data for each event? Not getting the 'fields' set up right for logstash?

I just tried it, and yes, all the fields are just concated into one message with a timestamp etc.
I would very much like to use the fields and send a formatted message to logstash.

I tried adding the 'codec' into the conf for filebeat, but that wasn't accepted, and the documentation is a bit hazy on this:
codec => multiline {
pattern => "-------------"
negate => false
what => "previous"
}

[fribse (Kenneth Fribert)]

I modified it a bit to this:
codec:
- multiline {
pattern => '^--*$'
negate => false
what => before
}
Now it's accepted in the config, but it doesn't help much.