# Grok filter for log files in logstash

**URL:** <https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377>\
**Category:** Logstash\
**Created:** [December 30, 2019, 7:03pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377 "2019-12-30T19:03:21Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [December 30, 2019, 7:03pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/1 "2019-12-30T19:03:22Z")

</div>

```auto
input {
  
  beats {
    port => 5044
  }
}

filter {
 if[log_type] =="access"{
    grok {
	match => {"message" => "%{COMBINEDAPACHELOG}"}
  } else if [log_type] == "errors" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
  }else [log_type] == "dispatcher" {
        grok {
            match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
        }
    }
}
 
output {
    elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    ilm_enabled => false
    index => "%{log_type}-%{+YYYY.MM.dd}"  
  }
  stdout {
    codec => rubydebug
  }
}

```

Below are the three log types I want to filter and extract information out of. Above is the file which has the match patterns.

```auto
Log Type 1: 08/10/2019 12:14:48 599 (null) DEBUG 27 GetUpdatedIncident for Incident Id 24749162 on thread

Log type 2: 08/10/2019 12:38:09 742 (null) DEBUG 10 Add activty in cache (152782646)

Log type 3: 2019-10-08 12:31:37,767 [pool-5-thread-47] INFO c.e.d.s.ScheduledActionProcessor - [24749750]EDI=NHA CustomFAULTSDEF: RR=NULL DispatchType=FLM RRDelay=0.0 RRThreshold=NULL DispatchWait=3 FaultSource=EMS HoldWhileServicing=false

```

I would like the Log type 3 to come out as below in kibana after the filter provided in filter block in logstash.conf but it isnt. Why?

```auto
{
  "threadId": "pool-5-thread-47",
  "logLevel": "INFO",
  "javaClass": "c.e.d.s.ScheduledActionProcessor",
  "incidentId": "24749750",
  "message": "EDI=NHA CustomFAULTSDEF: RR=NULL DispatchType=FLM RRDelay=0.0 RRThreshold=NULL DispatchWait=3 FaultSource=EMS HoldWhileServicing=false",
  "timestamp": "2019-10-08 12:31:37,767"
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 30, 2019, 8:35pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/2 "2019-12-30T20:35:05Z")

</div>

What does it come out like? What is the value of [log\_type]? Is there a \_grokparsefailure tag? Which index is the document in?

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [December 30, 2019, 8:43pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/3 "2019-12-30T20:43:50Z")

</div>

Hi @Badger  
It comes out as this-

```auto
Dec 30, 2019 @ 12:39:28.646

agent.ephemeral_id:
    00463acf-e130-4092-bb39-55fe18dbceaa
agent.hostname:
    mehak-VirtualBox
agent.version:
    7.4.0
agent.type:
    filebeat
agent.id:
    bad135c8-d359-4936-b515-79eb4bb24630
@version:
    1
host.name:
    mehak-VirtualBox
ecs.version:
    1.1.0
log.offset:
    41,656,653
log.file.path:
    /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log
fields.log_type:
    access
@timestamp:
    Dec 30, 2019 @ 12:39:28.646
tags:
    beats_input_codec_plain_applied
message:
    ExpectedFixDT: 08/10/2019 16:07,
type:
    another_test
_id:
    MM-

```

and after adding message tag, and fields.log\_type

```auto
Dec 30, 2019 @ 12:39:28.646 ExpectedFixDT: 08/10/2019 16:07,
access
Dec 30, 2019 @ 12:39:28.646 IncidentNo: 7928109745,
access
Dec 30, 2019 @ 12:39:28.647 ActivityDT: 08/10/2019 13:07,
access

```

The value of log\_type is defined in filebeat.yml file as follows:

```auto
filebeat.inputs:
- 
  paths:
     - /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log
  enabled: true
  input_type: log
  fields:  
    log_type: access

-
  paths:
     - /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log
  enabled: true
  input_type: log
  fields:  
     log_type: errors

-
  paths:
     - /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/dispatcher-log.log
  enabled: true
  input_type: log
  fields:  
     log_type: dispatch
  
output.logstash:
  hosts: ["localhost:5044"]

```

There is no \_grokparsefailure tag.

> [@Badger](#):
>
> Which index is the document in?

Do you mean where is it defined? Its is defined in logstash.conf

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 30, 2019, 9:39pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/4 "2019-12-30T21:39:04Z")

</div>

> [@Mehak\_Bhargava](#):
>
> if[log\_type] =="access"{

Your conditionals are testing a field called [log\_type]. However, your events have

```
fields.log_type:
access

```

a field called [fields][log\_type]. Either change the conditionals, or use the [fields\_under\_root](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#fields-under-root-log) option in filebeat.

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [December 30, 2019, 10:31pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/5 "2019-12-30T22:31:14Z")

</div>

```auto
filter {
 if[fields][log_type] =="access"{
    grok {
	match => {"message" => "%{COMBINEDAPACHELOG}"}
  } else if [fields][log_type] == "errors" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
  }else [fields][log_type] == "dispatcher" {
        grok {
            match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
        }
    }
}

```

Updated the [fields][log\_type] in file and now the output is below for Timestamp, message, fields.log\_type

```auto
Dec 30, 2019 @ 14:29:27.10508/10/2019 12:17:21 755 (null) INFO 24 Leftside Filter Expression : SubCategory="Servicing" for User ZKL15VT Item Count : 179
access

Dec 30, 2019 @ 14:29:27.11408/10/2019 12:17:22 012 (null) INFO 24 Leftside Filter Expression : SubCategory="Dispenser" AND SourceProblemName="Degraded" for User ZKL15VT Item Count : 63
access

```

But now errors and dispatcher logs arent showing up!

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [January 7, 2020, 1:26am UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/6 "2020-01-07T01:26:03Z")

</div>

Hi @Badger, even after fixing the conditional below, the required text isnt filtered properly. Any suggestions?

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [January 9, 2020, 10:11pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/7 "2020-01-09T22:11:00Z")

</div>

@Badger,

Is this fields tag right or should I add the fields\_under\_root option too?

```auto
input {
  
  beats {
    port => 5044
  }
}

filter 
{
 if[fields][log_type] =="access"
  {
    grok 
    {
	match => {"message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA} %{NONNEGINT:threadId}"}
    } 
  }else if [fields][log_type] == "errors" 
    {
        grok
        {
            match => { "message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA:message}" }
        }
  }
  else if [fields][log_type] == "dispatch" 
  {
        grok 
        {
            match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
        }
    }
}

output {
    elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    ilm_enabled => false
    index => "%[fields][log_type]-%{+YYYY.MM.dd}"  
  }
  stdout {
    codec => rubydebug
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2020, 10:24pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/8 "2020-01-09T22:24:53Z")

</div>

> [@Mehak\_Bhargava](#):
>
> Is this fields tag right or should I add the fields\_under\_root option too?

It is one or the other, it will not work if you do both.

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [January 9, 2020, 10:26pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/9 "2020-01-09T22:26:35Z")

</div>

I have not added the fields\_under\_root option and still the message isnt applying the grok pattern?  
Thanks, @Badger

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2020, 10:30pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/10 "2020-01-09T22:30:20Z")

</div>

You need to look at the data. I suggest you add

```
output { stdout { codec => rubydebug } }

```

and see what a dispatch event looks like.

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [January 9, 2020, 10:31pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/11 "2020-01-09T22:31:14Z")

</div>

> [@Mehak\_Bhargava](#):
>
> stdout { codec =\> rubydebug }

Is this not the same as you are suggesting? what is a dispatch event?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2020, 11:30pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/12 "2020-01-09T23:30:44Z")

</div>

By dispatch event I mean one in which you expect [fields][log\_type] to contain "dispatch".

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [January 9, 2020, 11:35pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/13 "2020-01-09T23:35:17Z")

</div>

currently, only access and error tagged logs are showing and not dispatch.

Also, I am considering adding multiple pipelines and make three config files with each file with its own filter and own output. [https://www.elastic.co/guide/en/logstash/7.5/multiple-pipelines.html](https://www.elastic.co/guide/en/logstash/7.5/multiple-pipelines.html)

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [January 10, 2020, 7:30pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/14 "2020-01-10T19:30:35Z")

</div>

@Badger, Made progress!!

after fixing the index error, Kibana actually enters the index name as

```auto
access-2020.01.10

```

which is how I wanted the index name to be.It is creating three different indexes now as expected. Thank you!Just needed to reload files

But the actual discussion is about grok pattern be used to filter logs which still isnt happening. Why is that happening any suggestions?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2020, 7:30pm UTC](https://discuss.elastic.co/t/grok-filter-for-log-files-in-logstash/213377/15 "2020-02-07T19:30:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.