# Grok with multiple matches - can i assign type based on match?

**URL:** <https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397>\
**Category:** Logstash\
**Created:** [September 13, 2016, 3:16pm UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397 "2016-09-13T15:16:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jamesmanning](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamesmanning/32/11882_2.png) [@jamesmanning](https://discuss.elastic.co/u/jamesmanning)\
**Post date:** [September 13, 2016, 3:16pm UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/1 "2016-09-13T15:16:12Z")

</div>

I'm parsing an IRC log and trying to assign different types based on which type of log entry a given line is.

Currently I'm using N different matches in a single grok filter, since a given line should match at most one of them, and then doing conditional mutates after. This feels like I'm doing something wrong.

I could break them up into N different grok calls each setting the type, but then I end up getting valid lines tagged as failures since each line will fail to match at least one of the expressions. I could override those to not set the failure tag, but it feels like I'm missing something going down this path.

I've tried adding add\_field after each match, but it then adds all the fields instead of just the one from the previous match.

Version:

```
iMac:logstash james$ logstash --version
logstash 2.4.0

```

Test input:

```
iMac:logstash james$ cat test.log 
--- Log opened Tue Sep 06 00:00:04 2016
--- Day changed Tue Sep 06 2016
00:00:04+0200 <+User1> Some message here
00:13:52+0200 -!- User2 [User2@some.site] has joined #somechannel
00:05:33+0200 -!- User3 [User3@some.other.site] has left #somechannel []
08:46:06+0200 * User4 does some action

```

Test configuration:

```
iMac:logstash james$ cat test.conf 
input {
  stdin { }
}

filter {
  grok {
    # normal chat message entry
    match => { "message" => "%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}%{SPACE}<[@ +*]%{SPACE}(?<user>[^>]+)>%{SPACE}(?<messageText>.*)" }

    # user did /me action
    match => { "message" => "%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}%{SPACE}\* (?<user>[^]+)%{SPACE}(?<action>.*)" }

    # user joined entry
    match => { "message" => "%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}%{SPACE}-!-%{SPACE}(?<user>[^]+) \[.*\] has joined .*" }

    # user left entry
    match => { "message" => "%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}%{SPACE}-!-%{SPACE}(?<user>[^]+) \[.*\] has left .*" }
  }

  if [messageText] =~ "." {
    mutate { replace => { "type" => "chatMessage" } }
  } else if [action] =~ "." {
    mutate { replace => { "type" => "action" } }
  } else if [message] =~ "has joined" {
    mutate { replace => { "type" => "joined" } }
  } else if [message] =~ "has left" {
    mutate { replace => { "type" => "left" } }
  }

  if "_grokparsefailure" in [tags] {
    drop {}
  }
}

output {
  stdout { codec => rubydebug }
}

```

Output from running shows it works fine, but that set of conditional mutate calls feels like I'm missing the Right Way(tm) to be doing this. 😄

```
iMac:logstash james$ logstash -f test.conf < test.log 
Settings: Default pipeline workers: 4
Pipeline main started
{
        "message" => "00:00:04+0200 <+User1> Some message here",
       "@version" => "1",
     "@timestamp" => "2016-09-13T15:06:31.485Z",
           "host" => "iMac.local",
           "user" => "User1",
    "messageText" => "Some message here",
           "type" => "chatMessage"
}
{
       "message" => "00:13:52+0200 -!- User2 [User2@some.site] has joined #somechannel",
      "@version" => "1",
    "@timestamp" => "2016-09-13T15:06:31.485Z",
          "host" => "iMac.local",
          "user" => "User2",
          "type" => "joined"
}
{
       "message" => "00:05:33+0200 -!- User3 [User3@some.other.site] has left #somechannel []",
      "@version" => "1",
    "@timestamp" => "2016-09-13T15:06:31.485Z",
          "host" => "iMac.local",
          "user" => "User3",
          "type" => "left"
}
{
       "message" => "08:46:06+0200 * User4 does some action",
      "@version" => "1",
    "@timestamp" => "2016-09-13T15:06:31.485Z",
          "host" => "iMac.local",
          "user" => "User4",
        "action" => "does some action",
          "type" => "action"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
iMac:logstash james$
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 13, 2016, 5:55pm UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/2 "2016-09-13T17:55:33Z")

</div>

Perhaps this would be more elegant:

```nohighlight
grok {
  # normal chat message entry
  match => { "message" => "%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}%{SPACE}<[@ +*]%{SPACE}(?<user>[^>]+)>%{SPACE}(?<messageText>.*)" }
  add_field => {
    "type" => "chatMessage"
  }
}

grok {
  # user did /me action
  match => { "message" => "%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}%{SPACE}\* (?<user>[^]+)%{SPACE}(?<action>.*)" }
  add_field => {
    "type" => "action"
  }
}

```

You'll have to deal with non-matches in some clever way, though. Perhaps by deleting the `type` field before the grok filters, setting `tag_on_failure => []` in all grok filters, and dropping the event unless `type` has been set?

---

<div class="post-metadata">

**Author:** ![jamesmanning](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamesmanning/32/11882_2.png) [@jamesmanning](https://discuss.elastic.co/u/jamesmanning)\
**Post date:** [September 13, 2016, 6:41pm UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/3 "2016-09-13T18:41:41Z")

</div>

That's a great idea! Thanks, @magnusbaeck !! 👍

---

<div class="post-metadata">

**Author:** ![lucianspec](https://avatars.discourse-cdn.com/v4/letter/l/3bc359/32.png) [@lucianspec](https://discuss.elastic.co/u/lucianspec)\
**Post date:** [June 30, 2017, 8:28am UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/4 "2017-06-30T08:28:05Z")

</div>

perhaps it's not a good suggestion...  
wrote two grok block will made the match execute twice everytime, it's total a waste.  
also break\_on\_match = true means grok filter break the match phase not the grok filter. so your example will get this

> "type" =\> ["chatMessage", "action"] most of time is not your want.

grok also had no way to get the match block result. so the only way i found is

> grok {  
> patterns\_dir =\> ["${PROJECT}/patterns"]  
> match =\> { "message" =\> "^%{STORM\_LOG:storm\_log}$" }  
> match =\> { "message" =\> "^%{STORM\_MESSAGE:storm\_message}$" }  
> }
> 
> ```
> if [storm_log] {
> mutate { 
> remove_field => "storm_log"
> add_field => { "match" => "STORM_LOG" }
> }
> } else if [storm_message] {
> mutate {
> remove_field => "storm_message"
> add_field => { "match" => "STORM_MESSAGE" }
> }
> }
> 
> ```

@magnusbaeck Right?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 30, 2017, 9:26am UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/5 "2017-06-30T09:26:33Z")

</div>

> so your example will get this

> "type" =\> ["chatMessage", "action"] most of time is not your want.

`add_field` only fires when the grok filter is successful. Unless both expressions match all messages I don't see why you'd get an array.

---

<div class="post-metadata">

**Author:** ![lucianspec](https://avatars.discourse-cdn.com/v4/letter/l/3bc359/32.png) [@lucianspec](https://discuss.elastic.co/u/lucianspec)\
**Post date:** [June 30, 2017, 10:27am UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/6 "2017-06-30T10:27:26Z")

</div>

okay, i read the source code and it's true.  
but i still want to point out it's hard to ensure string group each regex can recognize don't have duplicate elements.  
we adjust the priority by change the regex order.

> case match  
> when /integer/  
> puts "this is a integer"  
> when /int/  
> puts "this is a int"  
> when /i/  
> puts "this is a i"  
> end

most of time we need is implement a case statement, but seems grok's DSL made you have to choice:

1. edit branch logic but lose the "break" keyword
2. break\_on\_match but can't tell you which pattern is matched...

it's so strange...

---

<div class="post-metadata">

**Author:** ![Young-Bae\_Jeon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/young-bae_jeon/32/37961_2.png) [@Young-Bae\_Jeon](https://discuss.elastic.co/u/Young-Bae_Jeon)\
**Post date:** [November 22, 2018, 7:53am UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/7 "2018-11-22T07:53:05Z")

</div>

Thankyou!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!  
real thank you!!!!!!!!!!!!!!!!!!!!!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 4:28am UTC](https://discuss.elastic.co/t/grok-with-multiple-matches-can-i-assign-type-based-on-match/60397/8 "2022-11-04T04:28:11Z")

</div>