# Parsing message using Grok filter

**URL:** https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424
**Category:** Logstash
**Created:** [June 25, 2019, 7:40pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424 "2019-06-25T19:40:15Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 25, 2019, 7:40pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/1 "2019-06-25T19:40:15Z")

</div>

Hi,

I am trying to drop a couple of words in a message.

/test/data/user/log1/xyz

How can I drop the first two words "test" and "data" only when the message starts with "/test" and store username=user and file=log1/xyz?

Is it possible to write if else loop in grok filter? If message="/test/\*" use pattern A else use pattern B ?

I am trying to parse them using Grok filter but cannot come across the right REGEX for it

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 25, 2019, 8:34pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/2 "2019-06-25T20:34:29Z")

</div>

> [@naveenrt23](#):
>
> Is it possible to write if else loop in grok filter?

Yes, ruby [regexp](https://ruby-doc.org/core-2.5.3/Regexp.html#class-Regexp-label-Anchors) supports positive and negative lookahead and lookbehind assertions, as well as [alternation](https://ruby-doc.org/core-2.5.3/Regexp.html#class-Regexp-label-Alternation). This allows you to write complex if-elsif-elsif-else tests into your patterns. However, anyone who has to modify such a grok will probably re-write it from scratch.

You can do the test in the filter section using a conditional. For example

```
if [someField] =~ /^\/test\// {
      grok { match => [...]
} else {
      grok { match => [...]
}

```

If you are able to order your patterns such that only one applies then you can have grok try them all by supplying an array of patterns to the [match](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match) option.

You could start with

```
grok { match => { "someField" => "^/test/data/(?<user>[^/]+)/%{GREEDYDATA:fileName}" } }

```

---

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 26, 2019, 7:46pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/3 "2019-06-26T19:46:42Z")

</div>

Thanks for the info.

I have a single message which is composed of multiple values joined by pipes.  
`20190615|4|method|userend|/test/123/1.1|500|2`  
Now my condition would be "if the 5th value in a message starts with "/test" then use grok filter1 and else use grok filter2

I am trying to get the exact regex for it. Would the below work?

`^(.+?)\|(.+?)\|(.+?)\|(.+?)\|\/test\/.+?\|(.+?)\|.+$` . how can i use this in a condition to check if every message has /test as a 5th value ?

Thanks

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 26, 2019, 8:24pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/4 "2019-06-26T20:24:31Z")

</div>

grok is not the only tool in the toolbox.

```
    mutate { split => { "message" => "|" } }
    if [message][4] =~ /^\/test/ {
        [...]
    }
```

---

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 27, 2019, 2:13pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/5 "2019-06-27T14:13:53Z")

</div>

I've tried this pattern but I got grok parse failure

```
filter {
mutate { split => { "message" => "|" } }
    if [message][4] =~ /^\/test/ {

    grok {
            # Enable multiple matchers
            break_on_match => false

            match => { "message" => "%{DATA:timestamp_local}\|%{NUMBER:duration}\|%{WORD:requesttype}\|%{DATA:username}\|%{DATA:resource}\|%{NUMBER:statuscode}\|%{NUMBER:bytes}" }

            # Extract repo and path
            match => { "resource" => "/%{DATA:repo}/%{GREEDYDATA:resource_path}"}

            # Extract resource name
            match => { "resource_path" => "(?<resource_name>[^/]+$)" }
    }

}
}

```

Output:

`{"@version":"1","@timestamp":"2019-06-27T14:00:48.450Z","path":"/Users/testing/ai.log","host":"SI-M-C6G5","message":["20190615","4","method","userend","/test/123/1.1","500","2"],"tags":["_grokparsefailure"]}`

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 27, 2019, 2:37pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/6 "2019-06-27T14:37:24Z")

</div>

The split filter converts a string into an array. So after the split message looks like this:

```
   "message" => [
    [0] "20190615",
    [1] "4",
    [2] "method",
    [3] "userend",
    [4] "/test/123/1.1",
    [5] "500",
    [6] "2"

```

If you want to be able to grok the entire message field, then copy it to another field before splitting it

```
mutate { add_field => { "[@metadata][copyOfMessage]" => "%{[message]}" } }
mutate { split => { "[@metadata][copyOfMessage]" => "|" } }
if [@metadata][message][4] =~ /^\/test/ {
```

---

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 27, 2019, 3:20pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/7 "2019-06-27T15:20:39Z")

</div>

Thanks for the info. I apologize if I did not specify my requirement correctly. Just to reiterate

`20190615|4|method|userend|/test/123/1.1|500|2` If the 5th value in message starts with "/test", I need to drop it and store "/123/1.1" in a field.

The split filter does help me to split and check whether I have "/test" as the 5th value but how would I drop "/test" and store the rest in a field? I though grok would be the only way to do it.

```
filter {
mutate { add_field => { "[@metadata][copyOfMessage]" => "%{[message]}" } }
mutate { split => { "[@metadata][copyOfMessage]" => "|" } }
if [@metadata][message][4] =~ /^\/test/ {
grok {

        match => { "message" => "%{DATA:timestamp_local}\|%{NUMBER:duration}\|%{WORD:requesttype}\|%{DATA:username}\|%{DATA:resource}\|%{NUMBER:statuscode}\|%{NUMBER:bytes}" }

            # Extract repo and path
            match => { "resource" => "/%{DATA:repo}/%{GREEDYDATA:resource_path}"}

            # Extract resource name
            match => { "resource_path" => "(?<resource_name>[^/]+$)" }
    }
}
}

```

I've tried to copy the message in a new field and then parse it  
OUTPUT:

```
{
          "host" => "SI-M-C6G5",
      "@version" => "1",
    "@timestamp" => 2019-06-27T14:40:05.914Z,
       "message" => "20190615|4|method|userend|/test/123/1.1|500|2",
          "path" => "/Users/testing/ai.log"
}
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 27, 2019, 3:33pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/8 "2019-06-27T15:33:56Z")

</div>

> [@naveenrt23](#):
>
> 20190615|4|method|userend|/test/123/1.1|500|2

It works if you split it into 3 groks

```
    grok { match => { "message" => "%{DATA:timestamp_local}\|%{NUMBER:duration}\|%{WORD:requesttype}\|%{DATA:username}\|%{DATA:resource}\|%{NUMBER:statuscode}\|%{NUMBER:bytes}" } }
    grok { match => { "resource" => "/%{DATA:repo}/%{GREEDYDATA:resource_path}"} } }
    grok { match => { "resource_path" => "(?<resource_name>[^/]+$)" } }

```

---

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 27, 2019, 3:43pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/9 "2019-06-27T15:43:10Z")

</div>

```
filter {
mutate { add_field => { "[@metadata][copyOfMessage]" => "%{[message]}" } }
mutate { split => { "[@metadata][copyOfMessage]" => "|" } }
if [@metadata][message][4] =~ /^\/test/ {
grok { match => { "message" => "%{DATA:timestamp_local}\|%{NUMBER:duration}\|%{WORD:requesttype}\|%{DATA:username}\|%{DATA:resource}\|%{NUMBER:statuscode}\|%{NUMBER:bytes}" } }
grok { match => { "resource" => "/(?<repo>[^\/]+)/%{GREEDYDATA:resource_path}"} }
grok { match => { "resource_path" => "(?<resource_name>[^/]+$)" } }
}
}

```

I've tried using multiple groks but still see the same output

```
{
          "host" => "SI-M-C6G5",
      "@version" => "1",
    "@timestamp" => 2019-06-27T15:36:34.446Z,
       "message" => "20190615|4|method|userend|/test/123/1.1|500|2",
          "path" => "/Users/testing/ai.log"
}
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 27, 2019, 3:47pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/10 "2019-06-27T15:47:47Z")

</div>

> [@naveenrt23](#):
>
> if [@metadata][message][4] =~

Replace [message] with [copyOfMessage]

---

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 27, 2019, 9:04pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/11 "2019-06-27T21:04:06Z")

</div>

Thanks a lot @Badger

---

<div class="post-metadata">

### Author: ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)
#### Post date: [June 28, 2019, 2:03pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/12 "2019-06-28T14:03:07Z")

</div>

Just a small question,

Why does splitting groks and mutate filters work and not if when all the commands are in a single filter ?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 28, 2019, 2:21pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/13 "2019-06-28T14:21:26Z")

</div>

> [@naveenrt23](#):
>
> Why does splitting groks and mutate filters work and not if when all the commands are in a single filter ?

I am not sure. I do know that specifying the same option to a filter multiple times often works, but sometimes does not. It is very confusing, so I avoid doing it.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 26, 2019, 2:21pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424/14 "2019-07-26T14:21:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.