# Grok filter for logstash/filebeats

**URL:** <https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481>\
**Category:** Logstash\
**Created:** [January 24, 2020, 7:32pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481 "2020-01-24T19:32:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 24, 2020, 7:32pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/1 "2020-01-24T19:32:58Z")

</div>

I'm trying to extract my logs field using grok but it doesn't work.

Here's my grok filter:

filter { # parse the CSV structure generated from the log file into fields grok { match =\> {"message" =\> "%{GREEDYDATA:Job\_Name} %{HAPROXYTIME:ElapsedRenderTime} %{NUMBER:FrameCount} %{WORD:FrameList} %{WORD:Render\_Type} %{WORD:UserSubmitter} %{HAPROXYTIME:TotalTaskRenderTime} %{HAPROXYTIME:JobAverageFrameRenderTime} %{DATESTAMP:SubmitDate} %{DATESTAMP:CompletedDate} %{WORD:MachineSubmitted} %{NUMBER:JobPriority} %{WORD:Job\_ID}" } } }

Here's my logfile:

Job\_Name: TESTj  
ElapsedRenderTime: 00:00:27.8370000  
FrameCount: 1  
FrameList: 21  
Render\_Type: AfterEffects  
UserSubmitter: user1  
TotalTaskRenderTime: 00:00:23.2210000  
JobAverageFrameRenderTime: 00d:00h:00m:23s  
SubmitDate: 01/24/2020 09:56:17  
CompletedDate: 01/24/2020 09:56:47  
MachineSubmitted: D1-RN-XX01  
JobPriority: 50  
Job\_ID: 5e2b2fc1dddcae24084784b7

I've also tried %{WORD:Job\_Name} but the result is this:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/2/82e729460c52b87058d589064753cfceceb8c599.png)

The result is incorrect. This should have been "TESTj" instead of "Job\_Name".

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 25, 2020, 4:00pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/2 "2020-01-25T16:00:20Z")

</div>

I would have grok match that against a list of patterns. If you are using a multiline filter to combine the logfile into a single event you will need 'break\_on\_match =\> false'

```
grok {
    match => {
        "message" => [
            "Job_Name: %{WORD:Job_Name}",
            "ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
            "FrameCount: %{NUMBER:FrameCount}",
            ...
        ]
    }
}

```

I am not convinced that 00:00:27.8370000 matches the HAPROXYTIME pattern.

---

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [January 25, 2020, 6:17pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/3 "2020-01-25T18:17:07Z")

</div>

I would recommend trying kv filter instead of grok.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 4:29pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/4 "2020-01-27T16:29:57Z")

</div>

Does "break\_on\_match" would go under file or filter section? I have used the grok debugger online and HAPROXYTIME is what it showed for ElapsedRenderTime filed.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 4:30pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/5 "2020-01-27T16:30:17Z")

</div>

How would I use kv filter? thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 27, 2020, 5:35pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/6 "2020-01-27T17:35:33Z")

</div>

You would add

```
break_on_match => false

```

to the grok filter.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 5:36pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/7 "2020-01-27T17:36:29Z")

</div>

does it require a pattern directory?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 27, 2020, 5:47pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/8 "2020-01-27T17:47:14Z")

</div>

No, it does not.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 5:50pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/9 "2020-01-27T17:50:38Z")

</div>

filter {  
# parse the CSV structure generated from the log file into fields

```
break_on_match => false
grok { match => { "message" => [ "Job_Name: %{WORD:Job_Name}",
								"ElapsedRenderTime: %HAPROXYTIME:ElapsedRenderTime}", 
								"FrameCount: %{NUMBER:FrameCount}",
								"FrameList: %{WORD:FrameList}",
								"Render_Type: %{WORD:Render_Type}",
								"UserSubmitter: %{WORD:UserSubmitter}",
								"TotalRenderTime: %{HAPROXYTIME:TotalRenderTime}",
								"JobAverageFrameRenderTime: %{HAPROXYTIME:JobAverageFrameRenderTime}",
								"SubmitDate: %{TIMESTAMP:SubmitDate}",
								"CompletedDate: %{TIMESTAMP:CompletedDate}",
								"MachineSubmitted: %{WORD:MachineSubmitted}",
								"JobPriority: %{NUMBER:JobPriority}",
								"Job_ID: %{WORD:Job_ID}" ] }

	}

```

Here's my grok filter but running the logstash service shows successful but it shutdown. I'm also running filebeat but logstash.yml is disabled.  
I still don't see the result my looking for.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 27, 2020, 6:04pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/10 "2020-01-27T18:04:53Z")

</div>

The break\_on\_match option has to be _inside_ the grok filter.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 6:11pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/11 "2020-01-27T18:11:05Z")

</div>

Here's my complete config:

input {  
file {  
path =\> "\d1motion-fs\kibana\_logs\render\_logs\*.txt"  
start\_position =\> "beginning"  
}  
}

filter {  
grok { break\_on\_match =\> false  
match =\> { "message" =\> [ "Job\_Name: %{WORD:Job\_Name}",  
"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",  
"FrameCount: %{NUMBER:FrameCount}",  
"FrameList: %{WORD:FrameList}",  
"Render\_Type: %{WORD:Render\_Type}",  
"UserSubmitter: %{WORD:UserSubmitter}",  
"TotalRenderTime: %{HAPROXYTIME:TotalRenderTime}",  
"JobAverageFrameRenderTime: %{HAPROXYTIME:JobAverageFrameRenderTime}",  
"SubmitDate: %{TIMESTAMP:SubmitDate}",  
"CompletedDate: %{TIMESTAMP:CompletedDate}",  
"MachineSubmitted: %{WORD:MachineSubmitted}",  
"JobPriority: %{NUMBER:JobPriority}",  
"Job\_ID: %{WORD:Job\_ID}" ] }

```
	}

```

}  
output {

```
elasticsearch {
	hosts => ["D1motion-fs:9200"]
	
}

```

}

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 6:11pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/12 "2020-01-27T18:11:43Z")

</div>

Please let me know if there's any error on this config. I'm still learning this thing.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 27, 2020, 6:11pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/13 "2020-01-27T18:11:58Z")

</div>

> [@gsuboc](#):
>
> path =\> "\d1motion-fs\kibana\_logs\render\_logs\*.txt"

Do not use backslash in the path option of a file input. Use forward slash.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 6:13pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/14 "2020-01-27T18:13:00Z")

</div>

It's a windows platform. I have tried the // but it doesn't grab my files from that location.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 6:25pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/15 "2020-01-27T18:25:59Z")

</div>

It's actually this:

path =\> "\\d1motion-fs\kibana\_logs\render\_logs\*.txt"

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 27, 2020, 6:29pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/16 "2020-01-27T18:29:52Z")

</div>

> [@gsuboc](#):
>
> path =\> "\d1motion-fs\kibana\_logs\render\_logs\*.txt"

As I said, do not use backslash. Try

```
path => "/d1motion-fs/kibana_logs/render_logs/*.txt"

```

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 6:34pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/17 "2020-01-27T18:34:25Z")

</div>

Changed path to:  
path =\> "//d1motion-fs/kibana\_logs/render\_logs/\*.txt"

I'll run some test and will post the result, thanks.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 6:38pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/18 "2020-01-27T18:38:26Z")

</div>

Here's my result:

log.flags multiline  
log.offset 0  
message Job\_Name: TESTq  
ElapsedRenderTime: 00:00:28.8190000  
FrameCount: 1  
FrameList: 8  
Render\_Type: AfterEffects  
UserSubmitter: gssuboc  
TotalTaskRenderTime: 00:00:24.3090000  
JobAverageFrameRenderTime: 00d:00h:00m:24s  
SubmitDate: 01/27/2020 10:34:42  
CompletedDate: 01/27/2020 10:35:12  
MachineSubmitted: D1-RN-XX01  
JobPriority: 50

The fields are still under the "message" section, not separated.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 27, 2020, 7:11pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/19 "2020-01-27T19:11:29Z")

</div>

What do you get when you use

```
output { stdout { codec => rubydebug } }

```

?

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 27, 2020, 7:12pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/20 "2020-01-27T19:12:53Z")

</div>

same result as above.

[Next page](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481.md?page=2)