# Grok filter for logstash/filebeats

**URL:** <https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481>\
**Category:** Logstash\
**Created:** [January 24, 2020, 7:32pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481 "2020-01-24T19:32:57Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 28, 2020, 8:23pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/21 "2020-01-28T20:23:15Z")

</div>

any recommendation?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 28, 2020, 10:04pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/22 "2020-01-28T22:04:36Z")

</div>

You didn't answer my question, so I cannot add anything.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 28, 2020, 11:36pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/23 "2020-01-28T23:36:48Z")

</div>

I'm getting the same result with output { stdout { codec =\> rubydebug } }.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 29, 2020, 12:40am UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/24 "2020-01-29T00:40:34Z")

</div>

I am asking you to show me what that rubydebug codec produces.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 29, 2020, 1:48am UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/25 "2020-01-29T01:48:05Z")

</div>

Here’s the result:

 ![image001.png](https://us1.discourse-cdn.com/elastic/original/3X/2/c/2c5ab9c64bf7c7e63ffd26b64337e88a3f224c0a.png)

---

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [January 29, 2020, 7:41am UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/26 "2020-01-29T07:41:32Z")

</div>

Ok, using this log:

```
Job_Name: TESTj
ElapsedRenderTime: 00:00:27.8370000
FrameCount: 1
FrameList: 21
Render_Type: AfterEffects
UserSubmitter: user1
TotalTaskRenderTime: 00:00:23.2210000
JobAverageFrameRenderTime: 00d:00h:00m:23s
SubmitDate: 01/24/2020 09:56:17
CompletedDate: 01/24/2020 09:56:47
MachineSubmitted: D1-RN-XX01
JobPriority: 50
Job_ID: 5e2b2fc1dddcae24084784b7

```

and this pipeline:

```
input {
	file {
		path => "/tmpBIS/ror/elastic_env/logstash-6.7.1/testlog"
		start_position => "beginning"
 		sincedb_path => "/dev/null"
	}
}

filter {
	grok { 
		break_on_match => true
		match => { "message" => [ 
			"Job_Name: %{WORD:Job_Name}",
			"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
			"FrameCount: %{NUMBER:FrameCount}",
			"FrameList: %{NUMBER:FrameList}",
			"Render_Type: %{WORD:Render_Type}",
			"UserSubmitter: %{WORD:UserSubmitter}",
			"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",
			"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",
			"SubmitDate: %{DATESTAMP:SubmitDate}",
			"CompletedDate: %{DATESTAMP:CompletedDate}",
			"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",
			"JobPriority: %{NUMBER:JobPriority}",
			"Job_ID: %{WORD:Job_ID}" ] }
		}

	mutate {
		remove_field => ["message","host","@version","path","@timestamp","haproxy_minute","haproxy_hour","haproxy_second"]
	}
}
output {
	stdout {codec=> rubydebug}
}

```

Will result to :

```
{
    "FrameList" => "21"
}
{
    "JobAverageFrameRenderTime" => "00d:00h:00m:23s"
}
{
    "JobPriority" => "50"
}
{
    "FrameCount" => "1"
}
{
    "TotalTaskRenderTime" => "00:00:23.2210000"
}
{
    "MachineSubmitted" => "D1-RN-XX01"
}
{
    "Job_Name" => "TESTj"
}
{
    "Render_Type" => "AfterEffects"
}
{
    "SubmitDate" => "01/24/2020 09:56:17"
}
{
    "ElapsedRenderTime" => "00:00:27.8370000"
}
{
    "UserSubmitter" => "user1"
}
{
    "CompletedDate" => "01/24/2020 09:56:47"
}
```

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 29, 2020, 4:47pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/27 "2020-01-29T16:47:42Z")

</div>

Thanks for replying. Here’s the latest result with your current config.

 ![image001.png](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e09a69bce02f6e215e265c4992c9eda22a09d68.png)

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 29, 2020, 4:48pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/28 "2020-01-29T16:48:44Z")

</div>

Here’s the config file:

input {

file {

path =\> "\D1motion-fs\kibana\_logs\render\_logs\render\*.log"

start\_position =\> "beginning"

sincedb\_path =\> "/dev/null"

}

}

filter {

grok { break\_on\_match =\> false

match =\> { "message" =\> [

"Job\_Name: %{WORD:Job\_Name}",

"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",

"FrameCount: %{NUMBER:FrameCount}",

"FrameList: %{NUMBER:FrameList}",

"Render\_Type: %{WORD:Render\_Type}",

"UserSubmitter: %{WORD:UserSubmitter}",

"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",

"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",

"SubmitDate: %{DATESTAMP:SubmitDate}",

"CompletedDate: %{DATESTAMP:CompletedDate}",

"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",

"JobPriority: %{NUMBER:JobPriority}",

"Job\_ID: %{WORD:Job\_ID}" ] }

}

mutate {

remove\_field =\> ["message","host","@version","path","@timestamp","haproxy\_minute","haproxy\_hour","haproxy\_second"]

}

}

output {

stdout { codec =\> rubydebug }

}

 ![image001.png](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e09a69bce02f6e215e265c4992c9eda22a09d68.png)

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [January 30, 2020, 7:49pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/29 "2020-01-30T19:49:06Z")

</div>

Still not separating the fieldnames from the log.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 3, 2020, 5:19pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/30 "2020-02-03T17:19:50Z")

</div>

Any other recommendation? Thanks.

---

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [February 4, 2020, 3:19pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/31 "2020-02-04T15:19:40Z")

</div>

can you post your filebeat config? looks like your filebeat is reading your logs using multiline patterns

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 4, 2020, 4:29pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/32 "2020-02-04T16:29:46Z")

</div>

Here you go.

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 4, 2020, 5:16pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/33 "2020-02-04T17:16:41Z")

</div>

I’ve commented the multiline for filebeat and here’s the result. It still shows under message.

 ![image001.png](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f624ec87e2be1aa98d4f72d9b72d21564ec01279.png)

 ![image002.png](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2d92e5f338956271afa601b3f3d6937de8078395.png)

---

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [February 4, 2020, 7:47pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/34 "2020-02-04T19:47:42Z")

</div>

You need to show us ur filebeat file, bcoz by default, filebeat will read your file line per line unless you have enabled multiline pattern. I can see a multiline config in ur screenshot. You need to disable it. Then use my filter above. If you can’t provide us what we need then there’s no way we can help you.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 4, 2020, 8:01pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/35 "2020-02-04T20:01:47Z")

</div>

Attached is the current filebeat file with the filter you have provided. Please let me know if you need other info.

Thanks.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e0085fb2b83b724b10ef35166e6763b1c9b1040.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/d/edaaa0422f887837462f58a3ff50810e5c7f648f.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/6/163543c58bd950c0bfd601c809322bc006a2ae8d.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/1/01a75f7582b21ad39a8ec64351698fb1923f800f.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/7856292c992c1002978c230a34172d8d3d4f3fa1.png)

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 5, 2020, 9:30pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/37 "2020-02-05T21:30:13Z")

</div>

Does this screenshot helps you out?

---

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [February 6, 2020, 1:15pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/38 "2020-02-06T13:15:53Z")

</div>

Yes, disable multiline. Its normal that it's under message field. Now, use grok filter to parse that out from message field.

```
grok { 
		break_on_match => true
		match => { "message" => [ 
			"Job_Name: %{WORD:Job_Name}",
			"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
			"FrameCount: %{NUMBER:FrameCount}",
			"FrameList: %{NUMBER:FrameList}",
			"Render_Type: %{WORD:Render_Type}",
			"UserSubmitter: %{WORD:UserSubmitter}",
			"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",
			"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",
			"SubmitDate: %{DATESTAMP:SubmitDate}",
			"CompletedDate: %{DATESTAMP:CompletedDate}",
			"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",
			"JobPriority: %{NUMBER:JobPriority}",
			"Job_ID: %{WORD:Job_ID}" ] }
		}
```

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 6, 2020, 2:43pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/39 "2020-02-06T14:43:27Z")

</div>

Yes, that’s what I have on my logstash but I can’t aggregate and visualize that field.

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 6, 2020, 10:55pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/40 "2020-02-06T22:55:13Z")

</div>

Is there a way to make my fields aggregatable?

---

<div class="post-metadata">

**Author:** ![gsuboc](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gsuboc](https://discuss.elastic.co/u/gsuboc)\
**Post date:** [February 10, 2020, 9:57pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481/41 "2020-02-10T21:57:03Z")

</div>

Do you have a recommendation? Thanks.

[Previous page](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481.md?page=1)

[Next page](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481.md?page=3)