# Multiple grok match don't work

**URL:** https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103
**Category:** Logstash
**Created:** [May 17, 2017, 12:41pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103 "2017-05-17T12:41:16Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)
#### Post date: [May 17, 2017, 12:41pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/1 "2017-05-17T12:41:16Z")

</div>

Hi, i have an ELK stack on docker and put logs througt "gelf", i need match logs by log format, i have two files (access.log and error.log, and one single tag "apache") i have this filter

```
filter {
    if [tag] == "apache" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
            add_field => ["logtype", "apache-log"]
        }
        grok {
            match => { "message" => "\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client %{IP:clientip}:.*\] %{DATA:errorcode}: %{GREEDYDATA:message}" }
            add_field => ["logtype", "apache-error"]
        }
    if "_grokparsefailure" in [tags] {
        drop {}
        }
    }
}

```

Only one grok works at a time, together no logs are available, if i comment first grok i have only errors if i comment second grok i have only access:

errors only:

```
filter {
    if [tag] == "apache" {
        # grok {
        # match => { "message" => "%{COMBINEDAPACHELOG}" }
        # add_field => ["logtype", "apache-log"]
        # }
        grok {
            match => { "message" => "\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client %{IP:clientip}:.*\] %{DATA:errorcode}: %{GREEDYDATA:message}" }
            add_field => ["logtype", "apache-error"]
        }
    if "_grokparsefailure" in [tags] {
        drop {}
        }
    }
}

```

access only:

```
filter {
    if [tag] == "apache" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
            add_field => ["logtype", "apache-log"]
        }
        # grok {
        # match => { "message" => "\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client %{IP:clientip}:.*\] %{DATA:errorcode}: %{GREEDYDATA:message}" }
        # add_field => ["logtype", "apache-error"]
        # }
    if "_grokparsefailure" in [tags] {
        drop {}
        }
    }
}

```

Is there a way to have them both?

---

<div class="post-metadata">

### Author: ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)
#### Post date: [May 17, 2017, 12:47pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/2 "2017-05-17T12:47:05Z")

</div>

you could add the remove\_tag option to the grok filter?

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-remove\_tag](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-remove_tag)

this will trigger on success.

---

<div class="post-metadata">

### Author: ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)
#### Post date: [May 17, 2017, 12:54pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/3 "2017-05-17T12:54:29Z")

</div>

Sorry but I do not understand how it can help me? an example?

---

<div class="post-metadata">

### Author: ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)
#### Post date: [May 17, 2017, 12:57pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/4 "2017-05-17T12:57:53Z")

</div>

> [@stefanocog](#):
>
> filter {  
> if [tag] == "apache" {  
> grok {  
> match =\> { "message" =\> "%{COMBINEDAPACHELOG}" }  
> add\_field =\> ["logtype", "apache-log"]  
> }  
> grok {  
> match =\> { "message" =\> "[(?\<timestamp\>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})] [%{WORD:module}:%{LOGLEVEL:loglevel}] [pid %{NUMBER:pid}:tid %{NUMBER:tid}] [client %{IP:clientip}:.\*] %{DATA:errorcode}: %{GREEDYDATA:message}" }  
> add\_field =\> ["logtype", "apache-error"]  
> }  
> if "\_grokparsefailure" in [tags] {  
> drop {}  
> }  
> }  
> }

Something like this.

```
filter {
    if [tag] == "apache" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
            add_field => ["logtype", "apache-log"]
	    remove_tag => ['_grokparsefailure']
        }
        grok {
            match => { "message" => "\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client %{IP:clientip}:.*\] %{DATA:errorcode}: %{GREEDYDATA:message}" }
            add_field => ["logtype", "apache-error"]
	    remove_tag => ['_grokparsefailure']
        }
    	if "_grokparsefailure" in [tags] {
          drop {}
        }
    }
}

```

---

<div class="post-metadata">

### Author: ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)
#### Post date: [May 17, 2017, 1:08pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/5 "2017-05-17T13:08:04Z")

</div>

In this way i have only error log...

---

<div class="post-metadata">

### Author: ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)
#### Post date: [May 17, 2017, 1:13pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/6 "2017-05-17T13:13:08Z")

</div>

Then COMBINEDAPACHELOG does not match.

you can test your grok patterns [here](http://grokdebug.herokuapp.com/)

---

<div class="post-metadata">

### Author: ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)
#### Post date: [May 17, 2017, 1:15pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/7 "2017-05-17T13:15:04Z")

</div>

But if i comment the error.log grok, i have access logs correctly....i have already test with grokdebug and work

---

<div class="post-metadata">

### Author: ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)
#### Post date: [May 17, 2017, 1:32pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/8 "2017-05-17T13:32:03Z")

</div>

If i invert order of grok i have only access log, like this

```
filter {
    if [tag] == "apache" {
        grok {
            match => { "message" => "\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client %{IP:clientip}:.*\] %{GREEDYDATA:message}" }
            add_field => ["logtype", "apache-error"]
        remove_tag => ["_grokparsefailure"]
        }
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
            add_field => ["logtype", "apache-log"]
        remove_tag => ["_grokparsefailure"]
        }
    if "_grokparsefailure" in [tags] {
        drop {}
        }
    }
}
```

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 22, 2017, 5:16am UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/9 "2017-05-22T05:16:06Z")

</div>

Why not set the `logtype` field already on the input side? Why do you need to use grok to figure out what kind of a log it is?

> ```
> if [tag] == "apache" {
> 
> ```

Do you really have a field named `tag`? Or did you mean `"apache" in [tags]`?

I suggest you disable the `_grokparsefailure` tag (using the `tag_on_failure` option) and change

```
if "_grokparsefailure" in [tags] {

```

into

```
if not [logtype] {

```

so that you delete events that haven't had the `logtype` field set, indicating that none of the grok filters matched.

But really, instead of dropping those events you should save them _somewhere_. How would you otherwise know if your grok filters are incorrectly failing to match some legitimate input events?

---

<div class="post-metadata">

### Author: ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)
#### Post date: [May 22, 2017, 7:56am UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/10 "2017-05-22T07:56:10Z")

</div>

I use gelf driver for docker, i can apply only "tag" or "label" ([https://docs.docker.com/engine/admin/logging/overview/#gelf](https://docs.docker.com/engine/admin/logging/overview/#gelf)), and can't apply tag to separate input file...  
i would like to install filebeat, but I have containers in alpine and I can not install it because in edge repository. I have (now) two types of file, apache access and apache errors, maybe you mean that way?

```
filter {
    if [tag] == "apache" {
        grok {
            match => { "message" => "\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client %{IP:clientip}:.*\] %{GREEDYDATA:message}" }
            add_field => ["logtype", "apache-error"]
        remove_tag => ["_grokparsefailure"]
        }
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
            add_field => ["logtype", "apache-log"]
        remove_tag => ["_grokparsefailure"]
        }
    if not [logtype] {
        drop {}
        }
    }
}

```

In any case the two patterns work perfectly, the problem is that one excludes the other

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 22, 2017, 12:32pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/11 "2017-05-22T12:32:30Z")

</div>

> I have (now) two types of file, apache access and apache errors, maybe you mean that way?

Yes, that looks reasonable except that I'd use `tag_on_failure` instead of `remove_tag`.

> In any case the two patterns work perfectly, the problem is that one excludes the other

Not sure what you mean by this.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 19, 2017, 12:32pm UTC](https://discuss.elastic.co/t/multiple-grok-match-dont-work/86103/12 "2017-06-19T12:32:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.