GROK Multiline Newby

Hi.
I have to GROK a multiline log.
It's around 15 lines per log.
Looking at some examples online, but they are a bit hard to follow....

Anybody got any real life examples?

I have installed windows filebeat and am ingesting the log files from there so maybe I can do some pre-formatting like from closest to the source?

TIA.

this config will merge any lines that start with whitespace to the previous line

filebeat:
  prospectors:
    -
      fields:
        cluster: mycluster
      paths:
        - "/som/path/log.log"
      input_type: log
      document_type: mongo
      multiline:
        pattern: '^[[:space:]]'
        negate: False
        match: after
  registry_file: /var/lib/filebeat/registry

Are you asking about how to use the multiline feature of Filebeat or how to use grok on the resulting events?

To get help, please post details including an example log entry.

Would prefer to do it in a central location --- Logstash Filter?
This is an example log - Mail Marshall logs.

The examples below are from this site:

**Receiving Logs:**
4020 14:17:16.960 Accepted 127.0.0.1:1300 on interface 127.0.0.1:25 (1 of 50 active), Total active 1. Assigning thread 3724
3724 14:17:16.960 DOS - IP whitelisted. <127.0.0.1> will not be evaluated.
3724 14:17:16.960 TX: <220 vm-example45 ESMTP MailMarshal (v6.9.5.2936) Ready>
3724 14:17:16.960 RX: <HELO vmexample45>
3724 14:17:16.960 TX: <250 vm-example45 Hello vmexample45 (127.0.0.1)>
3724 14:17:16.960 RX: <MAIL FROM: <admin@example.com>>
3724 14:17:16.960 TX: <250 sender ok <admin@example.com>>
3724 14:17:16.960 RX: <RCPT TO: <auser@example.com>>
3724 14:17:16.960 Checking user criteria for Rule Connection Policies:Deny Junk Mailers in Global Blacklist
3724 14:17:16.960 Created temp mail file <C:\Program Files (x86)\Marshal\MailMarshal\Queues\Incoming\~B4dae422c0000.000000000001.0001.mml>
3724 14:17:16.960 TX: <250 recipient ok <auser@example.com>>
3724 14:17:16.960 RX: <DATA>
3724 14:17:16.960 TX: <354 send the mail data, end with .>
3724 14:17:16.976 MessageId <<CF2AFCFD69974446B9FAEBE29579CCF9@vmexample45>> accepted for normal processing.
3724 14:17:16.976 Received Mail Message C:\Program Files (x86)\Marshal\MailMarshal\Queues\Incoming\B4dae422c0000.000000000001.0001.mml from 127.0.0.1, 1188 bytes.
3724 14:17:16.976 TX: <250 B4dae422c0000 Message accepted for delivery>
3724 14:17:16.976 RX: <QUIT>
3724 14:17:16.976 TX: <221 vm-example45 closing connection>
3724 14:17:16.976 Thread exiting for 127.0.0.1 after 16 millisecs

**Sending Logs:**
0336 14:17:17.007 Adding message  - Message ID:  age 20/04/11 14:17:16 to auser@example.com
0336 14:17:17.007 Starting thread 3160 for new message B4dae422c0000.000000000001.0001.mml 1343 bytes, Threads(New:1, Deferred:0, Static:1, DNS:0)
3160 14:17:17.007 Attempting delivery of B4dae422c0000.000000000001.0001.mml via route Local Domain - static route
3160 14:17:17.007 Delivering via Forward to IP IP(10.74.24.5):97
3160 14:17:17.007 RX: <220 example.com ESMTP Mail Service, Version: 42 -- ready at 04/20/11 14:17:17>
3160 14:17:17.007 TX: 
3160 14:17:17.023 RX: <250-example.com [10.74.24.5], this server offers 4 extensions>
3160 14:17:17.023 RX: <250-AUTH LOGIN>
3160 14:17:17.023 RX: <250-SIZE 5120000>
3160 14:17:17.023 RX: <250-HELP>
3160 14:17:17.023 RX: <250 AUTH=LOGIN>
3160 14:17:17.023 TX:  SIZE=1343>
3160 14:17:17.023 RX: <250 Requested mail action okay, completed>
3160 14:17:17.023 TX: >
3160 14:17:17.023 RX: <250 Requested mail action okay, completed>
3160 14:17:17.023 TX: 
3160 14:17:17.023 RX: <354 Start mail input; end with .>
3160 14:17:17.023 RX: <250 Requested mail action okay, completed>
3160 14:17:17.023 TX: 
3160 14:17:17.023 RX: <221 Service closing transmission channel>
3160 14:17:17.023 Thread exiting for B4dae422c0000.000000000001.0001.mml after 16 millisecs
0336 14:17:17.023 (1) Thread(s) has exited - 0 still active

Yes, and what do you want to do with these logs? What's the expected outcome?

Probably for now just be able to extract sender and receiver email address and then graph on them.

Okay, then you probably need to look into the aggregate filter in order to collect pieces of information from different events. It might be tempting to use a multiline filter to merge all lines for a given message into a single Logstash event but I wouldn't recommend it.