# Multiline grok of consecutive events of the same type with joining

**URL:** https://discuss.elastic.co/t/multiline-grok-of-consecutive-events-of-the-same-type-with-joining/172025
**Category:** Logstash
**Created:** [March 12, 2019, 7:11pm UTC](https://discuss.elastic.co/t/multiline-grok-of-consecutive-events-of-the-same-type-with-joining/172025 "2019-03-12T19:11:06Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Sous\_Lesquels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sous_lesquels/32/41843_2.png) [@Sous\_Lesquels](https://discuss.elastic.co/u/Sous_Lesquels)
#### Post date: [March 12, 2019, 7:11pm UTC](https://discuss.elastic.co/t/multiline-grok-of-consecutive-events-of-the-same-type-with-joining/172025/1 "2019-03-12T19:11:06Z")

</div>

Consider this log file structure:

```
patA1 .*? patA2 .*? patA3
... (many lines here)
patB1 .*? patB2 .*? patB3
patB1 .*? patB2 .*? patB3

```

In the above, consider `patXY` to be different regex patterns, i.e. the first line above could be `grok`ked via something like this:

```
grok {
  match => { 
    "message" => "%{patA1:keyA1} (?<afterA1:.*?) %{patA2:keyA2} (?<afterA2:.*?) %{patA3:keyA3}"}
  }
}

```

I'd like to `multiline` the above and then generate events from multiple lines as such:

```
event1: keyA3, keyB1, keyB3
event2: keyA3, keyB2, keyB3

```

That is, pick some part(s) from line A and add that to each of the B lines, ending up with as many events as there are Bs.

In other words, I'd like to process the above as if doing single-line processing of log lines such as:

```
patA1 .*? patA2 .*? patA3 .*? patB1 .*? patB2 .*? patB3
patA1 .*? patA2 .*? patA3 .*? patB1 .*? patB2 .*? patB3

```

from which I can pick items with grok alone.

Note that the lines are of the same structure, so when using `multiline`, the message for the above looks like:

```
"message" => "patA1 .*? patA2 .*? patA3\npatB1 .*? patB2 .*? patB3\npatB1 .*? patB2 .*? patB3"

```

i.e. it has A and all Bs in the same "block", since Bs are consecutive and all are of the same shape.

What's the best way to do this in logstash?

---

<div class="post-metadata">

### Author: ![Sous\_Lesquels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sous_lesquels/32/41843_2.png) [@Sous\_Lesquels](https://discuss.elastic.co/u/Sous_Lesquels)
#### Post date: [March 12, 2019, 7:59pm UTC](https://discuss.elastic.co/t/multiline-grok-of-consecutive-events-of-the-same-type-with-joining/172025/2 "2019-03-12T19:59:06Z")

</div>

First grok like this:

```
grok {
  match => {
    "message" => ".*?(?<keyA3>patA3}"
  }
}

```

This will add `keyA3` to the event. Then split:

```
split {}

```

This will split by `message` by default, giving you cloned events for each of the lines, thus keeping `keyA3` from this event. Then filter B lines only:

```
if ([message] !~ /patB1/) {
  drop {}
}

```

Then grok again for the rest of the needed info:

```
grok {
  match => {
    "message" => "(?<keyB1>patB1) .*? (?<keyB2>patB2) .*? (?<keyB3>patB3)"
  }
}

```

At this point all events will have `keyA3`, `keyB1`, `keyB2` and `keyB3` that you can use as needed.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 12, 2019, 8:26pm UTC](https://discuss.elastic.co/t/multiline-grok-of-consecutive-events-of-the-same-type-with-joining/172025/3 "2019-03-12T20:26:48Z")

</div>

That seems like a good approach provided the input file is small enough that you can handle it as a single multiline event.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 9, 2019, 8:26pm UTC](https://discuss.elastic.co/t/multiline-grok-of-consecutive-events-of-the-same-type-with-joining/172025/4 "2019-04-09T20:26:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.