Two nested Grok patterns not working

Elastic Stack Logstash
1

Hey everyone,

I'm new to using Logstash and Elasticsearch. I've been working on collecting logs through Filebeat and then using Logstash for parsing and data cleaning. I have two Grok patterns in place. The first one extracts some important fields, including a field called 'my_message' which holds a chunk of data(Gready Data). I'm then applying another Grok pattern on this 'my_message' field to extract specific attributes.

Individually, both Grok patterns work fine. However, when I try to use them together, some attributes don't get extracted as expected. I've always found this community to be really helpful, and I believe you can assist me with this too. Thanks a lot in advance!

input {
  beats {
    port => 5045
  }
}

filter {
  if "jicofo" in [tags] {
    grok {
      match => {
        "[event][original]" => "Jicofo %{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:my_log_level}: \[%{POSINT:my_process_id}\] (?:\[room=%{DATA:my_meeting_name}@%{DATA}(?:\s+meeting_id=%{UUID:my_meeting_id}(?:\s+participant=%{DATA:my_participant})?)?\] )?\[%{GREEDYDATA:my_message}\]"
      }
    }

    grok {
      match => {
        "my_message" => [
          "Received session-accept \"%{GREEDYDATA:received_message}\"",
          "Sending a queued source-add, %{GREEDYDATA:sources_from}",
          "sources=%{GREEDYDATA:sources_equals}",
          "sources from %{DATA:sources_from_brackets}:\s*\[%{GREEDYDATA:sources}\]",
          "Member joined:%{DATA:member_joined} stats-id=%{DATA:stats_id} audioMuted=%{DATA:audio_muted} videoMuted=%{DATA:video_muted} role=%{DATA:role} isJibri=%{DATA:is_jibri} isJigasi=%{DATA:is_jigasi} isTranscriber=%{DATA:is_transcriber}, room=%{DATA:room}",
          "Room destroyed with reason=%{DATA:room_destroyed_reason}"
        ]
      }
    }
  }
}

output {
  if [tags] and "jicofo" in [tags] {
    elasticsearch {
      hosts => ["elasticSearch_Host:9200"]
      index => "jicofo-%{+YYYY}"
    }
  }
}
2

Hello,

if i understood your problem correctly you would like to parse all patterns of the second grok and at the moment only one is parsed?

You will have to tell grok to not stop parsing on the first matched pattern:

Try to set this parameter to false and now all 6 pattern should be executed.

BR

3

Not exactly. Only a few patterns of the first grok match and the same for the second grok.

4

I think you are trying to do too much in your initial grok.

If I understand you correctly, you are trying to parse messages that have a fixed prefix, and optional and variable field in square brackets, followed by a variable message in square brackets. If that is wrong then I consider that proof that your initial grok is too complicated.

I would start with

        match => {
            "[event][original]" => "^Jicofo %{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:my_log_level}: \[%{POSINT:my_process_id}\] (\[%{GREEDYDATA:[@metadata][meetingDetails]}\] )?\[%{GREEDYDATA:my_message}\]"
        }

(Note that I anchored the pattern to start of line, this is a fail-fast optimisation.) Then use a second grok to parse the meetingDetails

    if [@metadata][meetingDetails] {
        grok {
            match => {
                "[@metadata][meetingDetails]" => "room=%{DATA:my_meeting_name}@%{DATA}(?:\s+meeting_id=%{UUID:my_meeting_id}(?:\s+participant=%{DATA:my_participant})?)?"
            }
        }
    }

Then your last grok is OK as-is, since you only want the first match. You might want to add

${GREEDYDATA}

as the last match in that grok to avoid a _grokparsefailure tag when [my_message] does not match any of the six patterns you have.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.