# 3 grok filter in same file

**URL:** <https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228>\
**Category:** Logstash\
**Created:** [August 20, 2018, 7:06pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228 "2018-08-20T19:06:41Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 20, 2018, 7:06pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/1 "2018-08-20T19:06:41Z")

</div>

Hi,  
I have installed filebeat on a windows server to send logs of filezilla server to logstash.  
At this moment, it works, I see my logs.  
But it seems, only one filter works.

Here my configuration

```
   grok {
     match => { "message" => "\(%{GREEDYDATA:id_filezilla}\) %{DATESTAMP:date_filezilla} - %{GREEDYDATA:compte_filezilla} \(%{IPV4:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}" }
     remove_field => "message"
    }

    grok {
     match => { "message" => "\(%{GREEDYDATA:id_filezilla}\) %{DATESTAMP:date_filezilla} - \(%{GREEDYDATA:compte_filezilla}\) \(%{IPV4:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}" }
     remove_field => "message"
    }

    grok {
     match => { "message" => "\(%{GREEDYDATA:id_filezilla}\) %{DATESTAMP:date_filezilla} - %{GREEDYDATA:compte_filezilla} %{IPV6:adresseip_filezilla}> %{GREEDYDATA:action_filezilla}" }
     remove_field => "message"
    }

```

The first groks works, I have my fields with information.

The 2 others, they are no working. Specially the third, it's the same as the first but for ipv6 address.

Is my config fine ?

Or how i can improve it to have my 3 filters ?

thank you for your help.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 20, 2018, 7:13pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/2 "2018-08-20T19:13:03Z")

</div>

Comment out the `remove_field` lines and show us an example of an event that wasn't processed correctly. Use a `stdout { codec => rubydebug }` output to dump the raw event.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 20, 2018, 7:41pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/3 "2018-08-20T19:41:13Z")

</div>

Thnak you for your reply !

My output is the following :

> output {  
> elasticsearch {  
> hosts =\> "server:9200"  
> manage\_template =\> false  
> index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
> }  
> }

Can i add the output you give me with mine ?  
And how i can read the stdout ouput ? I don't know to see this with command line.

I see all my results in kibana, it's here where I have seen my results with the tag "\_grokparsefailure"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 20, 2018, 8:14pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/4 "2018-08-20T20:14:38Z")

</div>

> Can i add the output you give me with mine ?

Yes.

> I see all my results in kibana, it's here where I have seen my results with the tag "\_grokparsefailure"

You can copy/paste the raw event from Kibana's JSON tab, that's fine too.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 21, 2018, 10:07am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/5 "2018-08-21T10:07:03Z")

</div>

> [@magnusbaeck](#):
>
> stdout { codec =\> rubydebug }

Thank you.

I have added the new output.

Here the results :

it's OK with the following (first grok filter for an ipv4 address :  
I have my fields

> {  
> "\_index": "filebeat-6.3.0-2018.08.21",  
> "\_type": "doc",  
> "\_id": "CAbdW2UB-4GQqcKpWxkJ",  
> "\_version": 1,  
> "\_score": null,  
> "\_source": {  
> "message": "(011418) 21/08/2018 11:44:07 - account\_fz (IP\_SERVER)\> RETR BL\_2951519.PCL",  
> "host": {  
> "name": "SERVERNAME"  
> },  
> "prospector": {  
> "type": "log"  
> },  
> "input": {  
> "type": "log"  
> },  
> "offset": 2365486,  
> "@timestamp": "2018-08-21T09:44:07.938Z",  
> "@version": "1",  
> "id\_filezilla": "011418",  
> "date\_filezilla": "21/08/2018 11:44:07",  
> "source": "C:\Program Files (x86)\FileZilla Server\Logs\fzs-2018-08-21.log",  
> "tags": [  
> "beats\_input\_codec\_plain\_applied",  
> "\_grokparsefailure"  
> ],  
> "compte\_filezilla": "account\_fz",  
> "beat": {  
> "version": "6.3.0",  
> "name": "SERVERNAME",  
> "hostname": "SERVERNAME"  
> },  
> "action\_filezilla": "RETR BL\_2951519.PCL",  
> "adresseip\_filezilla": "172.21.5.28"  
> },  
> "fields": {  
> "@timestamp": [  
> "2018-08-21T09:44:07.938Z"  
> ]  
> },  
> "highlight": {  
> "adresseip\_filezilla": [  
> "@kibana-highlighted-field@172.21.5.28@/kibana-highlighted-field@"  
> ],  
> "action\_filezilla": [  
> "@kibana-highlighted-field@RETR BL\_2951519.PCL@/kibana-highlighted-field@"  
> ],  
> "compte\_filezilla": [  
> "@kibana-highlighted-field@ftp\_penta\_esker@/kibana-highlighted-field@"  
> ]  
> },  
> "sort": [  
> 1534844647938  
> ]  
> }

For the following, it's not ok, I don't have my fields, because, it's a ipv6 address (::1, localhost address) :  
It seems my grok filter for this don't work.

> {  
> "\_index": "filebeat-6.3.0-2018.08.20",  
> "\_type": "doc",  
> "\_id": "k8i0WGUB-4GQqcKpgfeC",  
> "\_version": 1,  
> "\_score": null,  
> "\_source": {  
> "tags": [  
> "beats\_input\_codec\_plain\_applied",  
> "\_grokparsefailure"  
> ],  
> "message": "(009668) 20/08/2018 21:00:29 - account\_fz (::1)\> STOR TRF\_CCOLLECT.004900",  
> "offset": 5923185,  
> "input": {  
> "type": "log"  
> },  
> "host": {  
> "name": "SERVERNAME"  
> },  
> "@version": "1",  
> "source": "C:\Program Files (x86)\FileZilla Server\Logs\fzs-2018-08-20.log",  
> "prospector": {  
> "type": "log"  
> },  
> "beat": {  
> "name": "SERVERNAME",  
> "hostname": "SERVERNAME",  
> "version": "6.3.0"  
> },  
> "@timestamp": "2018-08-20T19:00:38.913Z"  
> },  
> "fields": {  
> "@timestamp": [  
> "2018-08-20T19:00:38.913Z"  
> ]  
> },  
> "highlight": {  
> "message": [  
> "(009668) 20/08/2018 21:00:29 - account\_fz (::1)\> STOR @kibana-highlighted-field@TRF\_CCOLLECT@/kibana-highlighted-field@.004900"  
> ]  
> },  
> "sort": [  
> 1534791638913  
> ]  
> }

So, what's wrong with my filters ?

I have tested with [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/), it's OK.

Do I have to put my grok filter for ipv6 in another file ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 21, 2018, 11:18am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/6 "2018-08-21T11:18:37Z")

</div>

That's right, there's something wrong with the grok filter. Debug it by reducing it to the smallest possible expression and make sure that works, then continue building the expression until it breaks.

It might be unrelated in this case, but you're using GREEDYDATA excessively. There are very few cases where it makes sense to have more than one DATA or GREEDYDATA pattern in the same grok expression.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 21, 2018, 11:52am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/7 "2018-08-21T11:52:56Z")

</div>

Thank you for the idea !

I have found a solution.  
The filter for ipv4 and ipv6 address is now combined in 1 filter.

> filter {  
> grok {  
> match =\> { "message" =\> "(%{GREEDYDATA:id\_filezilla}) %{DATESTAMP:date\_filezilla} - %{GREEDYDATA:compte\_filezilla} (%{IP:adresseip\_filezilla})\> %{GREEDYDATA:action\_filezilla}" }  
> }  
> }

It's ok with ipv4 and ipv6 addresse 🙂

If I put just this filter, it's OK, no error like "grokparsefailure".  
But if I add the other filter to not take the "(" character, I hace the error "grokparsefailure"/

So, How I can filter the information with the "(" character ?  
Sometimes, it appears, sometime no.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 21, 2018, 12:30pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/8 "2018-08-21T12:30:34Z")

</div>

`(\()?`matches `(` zero or one times, i.e. it an optional left parenthesis.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 21, 2018, 1:12pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/9 "2018-08-21T13:12:59Z")

</div>

I have done this :

> `\(%{NUMBER:id_filezilla}\) %{DATESTAMP:date_filezilla} - ?(\()%{GREEDYDATA:compte_filezilla}?(\)) \(%{IP:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}`

With this one filter, I have my 3 filters (one for ipv4, one for ipv6, one for the parenthesis

One last question  
you give this : `(\()?`

when I have test on grok debugger, it works only for left parenthesis, if I change for right parenthesis, it don't work.

What's exactly the differrence between `(\()?` and `?(\()`

thank you for your answers, you help me a a lot !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 21, 2018, 1:49pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/10 "2018-08-21T13:49:22Z")

</div>

> when I have test on grok debugger, it works only for left parenthesis, if I change for right parenthesis, it don't work.

Use `(\))?`.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 21, 2018, 2:44pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/11 "2018-08-21T14:44:28Z")

</div>

it's strange

`(\))?` does'nt work, I always see the right parenthesis when I have a information with this caracter.

No error grokparsefailure in kibana.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 21, 2018, 6:52pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/12 "2018-08-21T18:52:37Z")

</div>

What do you mean by "see the right parenthesis"?

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 22, 2018, 7:20am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/13 "2018-08-22T07:20:49Z")

</div>

Hi,  
I have this information sent by filebeat :

> (011948) 21/08/2018 16:06:20 - (not logged in) (192.168.1.1)\> USER ftp\_user

I would like to escape the 2 parenthesis before and after "not logged in" but the parenthesis dont' appears all the sime.

my filter is at this moment the following (and it works only for the left parenthesis)  
` grok { match => { "message" => "\(%{NUMBER:id_filezilla}\) %{DATESTAMP:date_filezilla} - (\()?%{GREEDYDATA:compte_filezilla}(\))? \(%{IP:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}" } }`

I have this part `(\()?%{GREEDYDATA:compte_filezilla}(\))?`

so, for the left parenthesis, it works, it always escape when the filter find a left parenthesis

but for the right parenthesis at the end, it's not working and I don't understand.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 25, 2018, 1:48pm UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/14 "2018-08-25T13:48:41Z")

</div>

The problem is probably that you're using GREEDYDATA. Can you give an example of a message without parentheses?

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [August 27, 2018, 7:24am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/15 "2018-08-27T07:24:30Z")

</div>

here an example

> (011982) 21/08/2018 16:23:31 - ftp\_user (172.16.16.20)\> 226 Successfully transferred "/Folder2/"

Actually, when a user is connecter, we don't have parentheses. Before the user is connected, we have

> (not logged in)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 27, 2018, 9:31am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/16 "2018-08-27T09:31:26Z")

</div>

I suggest you match that by looking for a parenthesized string (`\((?<fieldname>[^)]*)\)`) **or** a regular space-delimited string (`%{NOTSPACE:fieldname}`):

```
(\((?<fieldname>[^)]*)\)|%{NOTSPACE:fieldname})
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 24, 2018, 9:31am UTC](https://discuss.elastic.co/t/3-grok-filter-in-same-file/145228/17 "2018-09-24T09:31:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.