# Multiple Grok pattern filters arent filtering multiple logs in one logstash file
**URL:** https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654
**Category:** Logstash
**Created:** [January 10, 2020, 7:09pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654 "2020-01-10T19:09:02Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 10, 2020, 7:09pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/1 "2020-01-10T19:09:02Z")
Below is my pipeline.conf where I want the filter block to apply three separate grok patterns on three different log files. But currently, it isnt working. Should I add multiple pipelines to it? or create three different config files with one filter pattern each? Or is there another way to it?
```auto
input {
beats {
port => 5044
}
}
filter
{
if[fields][log_type] =="access"
{
grok
{
match => {"message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA} %{NONNEGINT:threadId}"}
}
}else if [fields][log_type] == "errors"
{
grok
{
match => { "message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA:message}" }
}
}
else if [fields][log_type] == "dispatch"
{
grok
{
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 11, 2020, 4:22am UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/2 "2020-01-11T04:22:40Z")
@Mehak_Bhargava
Please share the sample data.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 12, 2020, 1:46am UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/3 "2020-01-12T01:46:43Z")
Log for 1st grok-
```auto
08/10/2019 12:14:48 599 (null) DEBUG 27 GetUpdatedIncident for Incident Id 24749162 on thread 04fd1833-8275-46ff-816f-9acf0c1f7724:80759 on Thread 27
08/10/2019 12:14:48 600 (null) DEBUG 19 Updating cache with activity (152775689) Add Item:True Modify Item: False
08/10/2019 12:14:48 601 (null) DEBUG 67 Applying dynamic filter
```
Log file 2nd grok-
```auto
09/10/2019 12:38:09 741 (null) DEBUG 61 Filter :
08/10/2019 12:38:09 742 (null) DEBUG 10 Add activty in cache (152782646)
08/10/2019 12:38:09 758 (null) DEBUG 10 Add incident activity to cache
08/10/2019 12:38:09 774 (null) DEBUG 61 Leftside Filter Expression : IncidentType=1 AND StatusCode="01" for User ZKH481F
```
Right now message without filter is showing as -
```auto
08/10/2019 12:38:11 268 (null) INFO 62 Leftside Filter Expression : SubCategory="Cash Management" AND SourceProblemName="AOC ATM is Out of Cash per Servicer" for User NBK22RA Item Count : 2
```
Whereas I want the filter to be applied and show this in mesage on kibana-
```auto
{
"code": "774",
"anum": "61",
"StatusCode": "01",
"timestamp": "08/10/2019 12:38:09"
}
```
Log for 3rd grok-
```auto
2019-10-08 12:31:37,767 [pool-5-thread-47] INFO c.e.d.s.ScheduledActionProcessor - [24749750]EDI=NHA CustomFAULTSDEF: RR=NULL DispatchType=FLM RRDelay=0.0 RRThreshold=NULL DispatchWait=3 FaultSource=EMS HoldWhileServicing=false
2019-10-08 12:31:37,787 [pool-5-thread-77] INFO c.e.d.s.ScheduledActionProcessor - [24749536]QueryResult : {"partyDetails":[{"duration":{"baseValueMinutes":0},"template":{"id":10,"name":"EDI Template","description":"EDI Template","templateType":"other","jsonDefinition":"{\"body\":\"{\\\"MaxRetries\\\" : \\\"2\\\",\\\"Ttl\\\" : \\\"3600\\\"}\"}","tenantId":100,"channelId":77},"contactChannel":"EDI","atmSchedule":"AnyHours","level":"B. Service Team","contactType":"Notification","waitForNextContact":false,"contactMapping":"28-29-77-47-0","lifeCycle":"Close","users":[{"securityGroupName":null,"template":{"id":10,"name":"EDI Template","description":"EDI Template","templateType":"other","jsonDefinition":"{\"body\":\"{\\\"MaxRetries\\\" : \\\"2\\\",\\\"Ttl\\\" : \\\"3600\\\"}\"}","tenantId":100,"channelId":77},"lastName":null,"isAvailable":false,"address":null,"orgName":null,"nextAvailableTime":null,"timeZone":null,"userName":null,"userId":null,"orgId":"","firstName":null,"sequenceNo":2}]}]}
2019-10-08 12:31:37,774 [pool-5-thread-96] INFO c.e.d.s.s.SqlAdapterImpl - executeInternalAll;
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 12, 2020, 4:42pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/4 "2020-01-12T16:42:40Z")
Is this from logstash log?
I need the data that you are receiving from filebeat not the logstash output.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 13, 2020, 6:47pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/5 "2020-01-13T18:47:19Z")
@mancharagopan, this is the sample data that is in the files that filebeat is receiving.
Below is my filebeat.yml-
```auto
filebeat.inputs:
-
paths:
- /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log
enabled: true
input_type: log
fields:
log_type: access
-
paths:
- /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log
enabled: true
input_type: log
fields:
log_type: errors
-
paths:
- /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/dispatcher-log.log
enabled: true
input_type: log
fields:
log_type: dispatch
output.logstash:
hosts: ["localhost:5044"]
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 16, 2020, 3:57am UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/7 "2020-01-16T03:57:57Z")
This is filebeat configuration. I need the data in one of the log file.
---
### Author:  [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)
#### Post date: [January 16, 2020, 2:14pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/8 "2020-01-16T14:14:42Z")
> [@Mehak\_Bhargava](#):
>
> 09/10/2019 12:38:09 741 (null) DEBUG 61 Filter : 08/10/2019 12:38:09 742 (null) DEBUG 10 Add activty in cache (152782646) 08/10/2019 12:38:09 758 (null) DEBUG 10 Add incident activity to cache 08/10/2019 12:38:09 774 (null) DEBUG 61 Leftside Filter Expression : IncidentType=1 AND StatusCode="01" for User ZKH481F
You sure your problem is not the right spacing in your syntax? For example the missing space between the `if` and the bracket?
Because trying to replicate your scenario (on a 7.0.0 stack version) and having:
**`filebeat.yml`**
```
filebeat.inputs:
- type: log
enabled: true
paths:
- .../access.log
fields:
log_type: access
- type: log
enabled: true
paths:
- .../errors.log
fields:
log_type: errors
```
**logstash.conf**
```
input {
beats {
port => 5044
}
}
filter {
if [fields][log_type] == "access" {
mutate {
add_field => { "custom_type" => "access" }
}
} else if [fields][log_type] == "errors" {
mutate {
add_field => { "custom_type" => "errors" }
}
}
}
output { stdout{} }
```
**access.log**
`08/10/2019 12:14:48 601 (null) DEBUG 67 Applying dynamic filter`
**errors.log**
`08/10/2019 12:38:09 742 (null) ERRORS 10 Add activty in cache (152782646)`
I do have the right `custom_type` field in each of the log.
Can you post here the Logstash output without any filter? Simply taking the data from the beats input?
P.S. alternatively try something like `if "access" in [fields][log_type]`
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 16, 2020, 11:33pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/9 "2020-01-16T23:33:57Z")
It worked with this config-
```auto
input {
beats {
port => 5044
}
}
filter {
if[fields][log_type] =="access" {
grok {
break_on_match => false
match => {
"message" => [
"%{DATESTAMP:timestamp}%{SPACE}%{NONNEGINT:code}%{GREEDYDATA}%{LOGLEVEL}%{SPACE}%{NONNEGINT:anum}%{SPACE}%{GREEDYDATA:logmessage}",
"(?(?<=activity\s\()\d+)"
]
}
}
} else if [fields][log_type] == "errors" {
grok {
break_on_match => false
match => {
"message" => [
"%{DATESTAMP:timestamp}%{SPACE}%{NONNEGINT:code}%{GREEDYDATA}%{LOGLEVEL}%{SPACE}%{NONNEGINT:anum}%{SPACE}%{GREEDYDATA:logmessage}",
"(?(?<=StatusCode=\")\d+)"
]
}
}
} else if [fields][log_type] == "dispatch" {
grok {
break_on_match => false
match => {
"message" => [
"\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}(\[%{NONNEGINT:incidentId}])?%{GREEDYDATA:message}",
"(?(?<=scheduledActionList\s\[)[\d,\s]+)"
]
}
}
if "" in [scheduledActionList] {
mutate {
gsub => ["scheduledActionList", " ", ""]
split => {"scheduledActionList" => ","}
}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 13, 2020, 11:34pm UTC](https://discuss.elastic.co/t/multiple-grok-pattern-filters-arent-filtering-multiple-logs-in-one-logstash-file/214654/10 "2020-02-13T23:34:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.