Multiline grok of consecutive events of the same type with joining

[Sous_Lesquels]

Consider this log file structure:

patA1 .*? patA2 .*? patA3
... (many lines here)
patB1 .*? patB2 .*? patB3
patB1 .*? patB2 .*? patB3

In the above, consider patXY to be different regex patterns, i.e. the first line above could be grokked via something like this:

grok {
  match => { 
    "message" => "%{patA1:keyA1} (?<afterA1:.*?) %{patA2:keyA2} (?<afterA2:.*?) %{patA3:keyA3}"}
  }
}

I'd like to multiline the above and then generate events from multiple lines as such:

event1: keyA3, keyB1, keyB3
event2: keyA3, keyB2, keyB3

That is, pick some part(s) from line A and add that to each of the B lines, ending up with as many events as there are Bs.

In other words, I'd like to process the above as if doing single-line processing of log lines such as:

patA1 .*? patA2 .*? patA3 .*? patB1 .*? patB2 .*? patB3
patA1 .*? patA2 .*? patA3 .*? patB1 .*? patB2 .*? patB3

from which I can pick items with grok alone.

Note that the lines are of the same structure, so when using multiline, the message for the above looks like:

"message" => "patA1 .*? patA2 .*? patA3\npatB1 .*? patB2 .*? patB3\npatB1 .*? patB2 .*? patB3"

i.e. it has A and all Bs in the same "block", since Bs are consecutive and all are of the same shape.

What's the best way to do this in logstash?

[Sous_Lesquels]

First grok like this:

grok {
  match => {
    "message" => ".*?(?<keyA3>patA3}"
  }
}

This will add keyA3 to the event. Then split:

split {}

This will split by message by default, giving you cloned events for each of the lines, thus keeping keyA3 from this event. Then filter B lines only:

if ([message] !~ /patB1/) {
  drop {}
}

Then grok again for the rest of the needed info:

grok {
  match => {
    "message" => "(?<keyB1>patB1) .*? (?<keyB2>patB2) .*? (?<keyB3>patB3)"
  }
}

At this point all events will have keyA3, keyB1, keyB2 and keyB3 that you can use as needed.

[Badger]

That seems like a good approach provided the input file is small enough that you can handle it as a single multiline event.