# Multiple GROK filters not applying to log message

**URL:** <https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020>\
**Category:** Logstash\
**Tags:** elastic-stack-alerting\
**Created:** [November 16, 2018, 10:10am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020 "2018-11-16T10:10:46Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 16, 2018, 10:10am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/1 "2018-11-16T10:10:46Z")

</div>

I have created 2 grok filters to match log messages, but the first filter is only applying the second filter is not applying to all log messages

but the 2 filters parsing logs, I have created the first filter for loglevel and second one for my needed fileds

```
 match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\])) %{GREEDYDATA}" }

match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - %{DATA:method} processing time for transactionId : %{WORD:transactionid} documentType : %{WORD:document type} is %{INT:duration:int}" }

```

below are my log messages

```
2018-11-16 10:09:58.079 [INFO] from application in pool-3-thread-8 - Converting Document Authentication Response for transactionId : N9xnFCkW5Lpxa9tj to legacy api version : 1.1
2018-11-16 10:09:58.079 [INFO] from application in pool-3-thread-8 - LegacyDocumentAuthenticationResponseConversion processing time for transactionId : N9xnFCkW5Lpxa9tj documentType : License is 0 msec
2018-11-16 10:09:58.079 [INFO] from application in pool-3-thread-8 - Encrypting and Inserting Transaction Data into DB for transactionId : N9xnFCkW5Lpxa9tj
2018-11-16 10:09:58.080 [INFO] from application in pool-3-thread-8 - Inserting headshot into DB for transactionId : N9xnFCkW5Lpxa9tj
2018-11-16 10:09:58.187 [INFO] from application in pool-3-thread-8 - updateTransaction : Number of Documents modified in db for transactionId : 0
2018-11-16 10:09:58.187 [INFO] from application in pool-3-thread-8 - updateTransaction : merchantId : 5dce6959-b2c2-4890-bdf3-3e0d36662c4c inserted true for transactionId :N9xnFCkW5Lpxa9tj
2018-11-16 10:09:58.187 [INFO] from application in pool-3-thread-8 - Updating Accounting Data for transactionId : N9xnFCkW5Lpxa9tj
2018-11-16 10:09:58.188 [INFO] from application in pool-3-thread-8 - updateAccounts : Number of Documents modified in db for transactionId : 0
2018-11-16 10:09:58.188 [INFO] from application in pool-3-thread-8 - updateAccounts : requestType : Authentication inserted true for transactionId :N9xnFCkW5Lpxa9tj
2018-11-16 10:09:58.188 [INFO] from application in pool-3-thread-8 - Accounting Data updated for transactionId : N9xnFCkW5Lpxa9tj, requestType : Authentication
2018-11-16 10:09:58.188 [INFO] from application in pool-3-thread-8 - TransactionDataEncryptionAndInsertion processing time for transactionId : N9xnFCkW5pxa9tj documentType : License is 109 msec
2018-11-16 10:09:58.188 [INFO] from application in pool-3-thread-8 - AuthenticateDocument processing time for transactionId : N9xnFCkW5Lpxa9tj documentType : Licensemerchant : 5dce6959-b2c2-4890-bdf3-3e0d36662c4c is 23389 msec
```

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 16, 2018, 11:14am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/2 "2018-11-16T11:14:13Z")

</div>

@ashok9177, Your first grok filter is become generic and that part is common in all logs. So please try to change the order of your grok filter and it should work.

Thanks.

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 16, 2018, 11:18am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/3 "2018-11-16T11:18:20Z")

</div>

If i change order also first one is only working second filter not applying

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 16, 2018, 11:30am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/4 "2018-11-16T11:30:46Z")

</div>

@ashok9177, Can you please use the below patterns:

```auto
match => { "message" => ["%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:loglevel}\] %{DATA} - %{DATA:method} processing time for transactionId : %{WORD:transactionid} documentType : %{WORD:document type} is %{INT:duration} %{GREEDYDATA}", "%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:loglevel}\] %{GREEDYDATA:message}" ] }

```

Please restart logstash service after making the above changes.

Hope so above filter will work for you.

Thanks.

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 16, 2018, 11:54am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/5 "2018-11-16T11:54:58Z")

</div>

Thanks for your reply, may I know the cause, because i have to write 10 grok filters

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 16, 2018, 12:00pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/6 "2018-11-16T12:00:45Z")

</div>

@ashok9177, is it work fine for you?

If yes, then you can add multiple grok patterns in single message as i have added 2 in above filter. Please separate the patterns with , (comma). One more thing keep the below pattern at last:

```auto
%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:loglevel}\] %{GREEDYDATA:message}

```

If we use multiple match in single grok pattern then all logs will parse from first pattern. I also faced the same issue earlier.

Thanks.

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 16, 2018, 12:05pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/7 "2018-11-16T12:05:44Z")

</div>

yes, you'r filter is working for me. but adding 10 filters in single line doesn't look good. do you have any idea, why filters not working now?

the same way it worked before

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 16, 2018, 12:12pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/8 "2018-11-16T12:12:12Z")

</div>

@ashok9177, is all these logs are coming from single log file path?

If these are coming from different file path we can use different grok filter for all log file.

> [@ashok9177](#):
>
> but adding 10 filters in single line doesn't look good

i also using same thing at my end and its not creating and performance issue for logstash node.

> [@ashok9177](#):
>
> do you have any idea, why filters not working now?
> 
> the same way it worked before

I also tried multiple messages in single grok filter but that not worked at my end. If it were working earlier at your end then i don't know how it was working.

Thanks.

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 16, 2018, 12:15pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/9 "2018-11-16T12:15:40Z")

</div>

@ashok9177,

> [@ashok9177](#):
>
> %{TIMESTAMP\_ISO8601:timestamp} ([%{WORD:loglevel}])) %{GREEDYDATA}

Your above filter have small mistake and i have corrected that in my filter. You have used small parenthesis () at loglevel field. What the purpose of these ()?

Thanks.

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 16, 2018, 12:24pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/10 "2018-11-16T12:24:39Z")

</div>

What the purpose of these ()?

Don't know copied from some website, but as a single filter it is working and can you please suggest me how to exclude api key word in loglevel place , in some log message there api keyword , below is example

2

> 018-11-16 12:24:00,129 [api] [MainThread] [INFO] Saving to /home/ubuntu/temp/caffe\_demos\_uploads/9e59effd-9534-4934-b3d7-c9182a07d507image.jpg., [MainThread] [INFO] Saving to /home/ubuntu/temp/caffe\_demos\_uploads/9e59effd-9534-4934-b3d7-c9182a07d507image.jpg.

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 16, 2018, 12:38pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/11 "2018-11-16T12:38:17Z")

</div>

@ashok9177

> [@ashok9177](#):
>
> can you please suggest me how to exclude api key word in loglevel place

You can drop anything using \ as shown below:

```auto
%{TIMESTAMP_ISO8601:timestamp}\,\d+\s\[\w+\]

```

Thanks.

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 16, 2018, 12:43pm UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/12 "2018-11-16T12:43:24Z")

</div>

But i want pick INFO,ERROR,DEBUG words and exclude other word like api

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [November 19, 2018, 10:36am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/13 "2018-11-19T10:36:10Z")

</div>

if the logs are coming from different servers, and each servers has different field name, how to write if condition in grok filter?

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [November 23, 2018, 4:06am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/14 "2018-11-23T04:06:01Z")

</div>

@ashok9177

> [@ashok9177](#):
>
> But i want pick INFO,ERROR,DEBUG words and exclude other word like api

```auto
%{TIMESTAMP_ISO8601:timestamp}\,\d+\s\[\w+\]

```

The above pattren show how you can include and discard any field or word. It was only for your reference.

```auto
%{TIMESTAMP_ISO8601:timestamp}\,\d+\s\[\w+\]\s\[\w+\s\]\s\[%{WORD:loglevel}

```

You can extend above filter as per your log and requirement.

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2018, 4:06am UTC](https://discuss.elastic.co/t/multiple-grok-filters-not-applying-to-log-message/157020/15 "2018-12-21T04:06:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.