# Two nested Grok patterns not working

**URL:** <https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533>\
**Category:** Logstash\
**Created:** [August 24, 2023, 5:14am UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533 "2023-08-24T05:14:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Priyaansh\_Dwivedi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyaansh_dwivedi/32/121317_2.png) [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)\
**Post date:** [August 24, 2023, 5:14am UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533/1 "2023-08-24T05:14:32Z")

</div>

Hey everyone,

I'm new to using Logstash and Elasticsearch. I've been working on collecting logs through Filebeat and then using Logstash for parsing and data cleaning. I have two Grok patterns in place. The first one extracts some important fields, including a field called 'my\_message' which holds a chunk of data(Gready Data). I'm then applying another Grok pattern on this 'my\_message' field to extract specific attributes.

Individually, both Grok patterns work fine. However, when I try to use them together, some attributes don't get extracted as expected. I've always found this community to be really helpful, and I believe you can assist me with this too. Thanks a lot in advance!

```auto
input {
  beats {
    port => 5045
  }
}

filter {
  if "jicofo" in [tags] {
    grok {
      match => {
        "[event][original]" => "Jicofo %{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:my_log_level}: \[%{POSINT:my_process_id}\] (?:\[room=%{DATA:my_meeting_name}@%{DATA}(?:\s+meeting_id=%{UUID:my_meeting_id}(?:\s+participant=%{DATA:my_participant})?)?\] )?\[%{GREEDYDATA:my_message}\]"
      }
    }

    grok {
      match => {
        "my_message" => [
          "Received session-accept \"%{GREEDYDATA:received_message}\"",
          "Sending a queued source-add, %{GREEDYDATA:sources_from}",
          "sources=%{GREEDYDATA:sources_equals}",
          "sources from %{DATA:sources_from_brackets}:\s*\[%{GREEDYDATA:sources}\]",
          "Member joined:%{DATA:member_joined} stats-id=%{DATA:stats_id} audioMuted=%{DATA:audio_muted} videoMuted=%{DATA:video_muted} role=%{DATA:role} isJibri=%{DATA:is_jibri} isJigasi=%{DATA:is_jigasi} isTranscriber=%{DATA:is_transcriber}, room=%{DATA:room}",
          "Room destroyed with reason=%{DATA:room_destroyed_reason}"
        ]
      }
    }
  }
}

output {
  if [tags] and "jicofo" in [tags] {
    elasticsearch {
      hosts => ["elasticSearch_Host:9200"]
      index => "jicofo-%{+YYYY}"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Shaoranlaos](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Shaoranlaos](https://discuss.elastic.co/u/Shaoranlaos)\
**Post date:** [August 24, 2023, 7:58am UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533/2 "2023-08-24T07:58:37Z")

</div>

Hello,

if i understood your problem correctly you would like to parse all patterns of the second grok and at the moment only one is parsed?

You will have to tell grok to not stop parsing on the first matched pattern:

> **[Grok filter plugin | Logstash Reference \[8.9\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-break_on_match)**

Try to set this parameter to false and now all 6 pattern should be executed.

BR

---

<div class="post-metadata">

**Author:** ![Priyaansh\_Dwivedi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyaansh_dwivedi/32/121317_2.png) [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)\
**Post date:** [August 24, 2023, 9:05am UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533/3 "2023-08-24T09:05:47Z")

</div>

Not exactly. Only a few patterns of the first grok match and the same for the second grok.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 24, 2023, 3:54pm UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533/4 "2023-08-24T15:54:46Z")

</div>

I think you are trying to do too much in your initial grok.

If I understand you correctly, you are trying to parse messages that have a fixed prefix, and optional and variable field in square brackets, followed by a variable message in square brackets. If that is wrong then I consider that proof that your initial grok is too complicated.

I would start with

```
        match => {
            "[event][original]" => "^Jicofo %{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:my_log_level}: \[%{POSINT:my_process_id}\] (\[%{GREEDYDATA:[@metadata][meetingDetails]}\] )?\[%{GREEDYDATA:my_message}\]"
        }

```

(Note that I anchored the pattern to start of line, this is a fail-fast optimisation.) Then use a second grok to parse the meetingDetails

```
    if [@metadata][meetingDetails] {
        grok {
            match => {
                "[@metadata][meetingDetails]" => "room=%{DATA:my_meeting_name}@%{DATA}(?:\s+meeting_id=%{UUID:my_meeting_id}(?:\s+participant=%{DATA:my_participant})?)?"
            }
        }
    }

```

Then your last grok is OK as-is, since you only want the first match. You might want to add

```
${GREEDYDATA}

```

as the last match in that grok to avoid a \_grokparsefailure tag when [my\_message] does not match any of the six patterns you have.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 21, 2023, 3:55pm UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533/5 "2023-09-21T15:55:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.