Grok filter for logstash/filebeats

gsuboc · January 28, 2020, 8:23pm

any recommendation?

Badger · January 28, 2020, 10:04pm

You didn't answer my question, so I cannot add anything.

gsuboc · January 28, 2020, 11:36pm

I'm getting the same result with output { stdout { codec => rubydebug } }.

Badger · January 29, 2020, 12:40am

I am asking you to show me what that rubydebug codec produces.

gsuboc · January 29, 2020, 1:48am

Here’s the result:

inhinyera16 · January 29, 2020, 7:41am

Ok, using this log:

Job_Name: TESTj
ElapsedRenderTime: 00:00:27.8370000
FrameCount: 1
FrameList: 21
Render_Type: AfterEffects
UserSubmitter: user1
TotalTaskRenderTime: 00:00:23.2210000
JobAverageFrameRenderTime: 00d:00h:00m:23s
SubmitDate: 01/24/2020 09:56:17
CompletedDate: 01/24/2020 09:56:47
MachineSubmitted: D1-RN-XX01
JobPriority: 50
Job_ID: 5e2b2fc1dddcae24084784b7

and this pipeline:

input {
	file {
		path => "/tmpBIS/ror/elastic_env/logstash-6.7.1/testlog"
		start_position => "beginning"
 		sincedb_path => "/dev/null"
	}
}


filter {
	grok { 
		break_on_match => true
		match => { "message" => [ 
			"Job_Name: %{WORD:Job_Name}",
			"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
			"FrameCount: %{NUMBER:FrameCount}",
			"FrameList: %{NUMBER:FrameList}",
			"Render_Type: %{WORD:Render_Type}",
			"UserSubmitter: %{WORD:UserSubmitter}",
			"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",
			"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",
			"SubmitDate: %{DATESTAMP:SubmitDate}",
			"CompletedDate: %{DATESTAMP:CompletedDate}",
			"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",
			"JobPriority: %{NUMBER:JobPriority}",
			"Job_ID: %{WORD:Job_ID}" ] }
		}

	mutate {
		remove_field => [ "message","host","@version","path","@timestamp","haproxy_minute","haproxy_hour","haproxy_second" ]
	}
}
output {
	stdout {codec=> rubydebug}
}

Will result to :

{
    "FrameList" => "21"
}
{
    "JobAverageFrameRenderTime" => "00d:00h:00m:23s"
}
{
    "JobPriority" => "50"
}
{
    "FrameCount" => "1"
}
{
    "TotalTaskRenderTime" => "00:00:23.2210000"
}
{
    "MachineSubmitted" => "D1-RN-XX01"
}
{
    "Job_Name" => "TESTj"
}
{
    "Render_Type" => "AfterEffects"
}
{
    "SubmitDate" => "01/24/2020 09:56:17"
}
{
    "ElapsedRenderTime" => "00:00:27.8370000"
}
{
    "UserSubmitter" => "user1"
}
{
    "CompletedDate" => "01/24/2020 09:56:47"
}

gsuboc · January 29, 2020, 4:47pm

Thanks for replying. Here’s the latest result with your current config.

gsuboc · January 29, 2020, 4:48pm

Here’s the config file:

input {

file {

path => "\D1motion-fs\kibana_logs\render_logs\render*.log"

start_position => "beginning"

sincedb_path => "/dev/null"

}

filter {

grok { break_on_match => false

match => { "message" => [

"Job_Name: %{WORD:Job_Name}",

"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",

"FrameCount: %{NUMBER:FrameCount}",

"FrameList: %{NUMBER:FrameList}",

"Render_Type: %{WORD:Render_Type}",

"UserSubmitter: %{WORD:UserSubmitter}",

"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",

"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",

"SubmitDate: %{DATESTAMP:SubmitDate}",

"CompletedDate: %{DATESTAMP:CompletedDate}",

"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",

"JobPriority: %{NUMBER:JobPriority}",

"Job_ID: %{WORD:Job_ID}" ] }

}

mutate {

remove_field => [ "message","host","@version","path","@timestamp","haproxy_minute","haproxy_hour","haproxy_second" ]

}

output {

stdout { codec => rubydebug }

}

gsuboc · January 30, 2020, 7:49pm

Still not separating the fieldnames from the log.

gsuboc · February 3, 2020, 5:19pm

Any other recommendation? Thanks.

inhinyera16 · February 4, 2020, 3:19pm

can you post your filebeat config? looks like your filebeat is reading your logs using multiline patterns

gsuboc · February 4, 2020, 4:29pm

Here you go.

###################### Filebeat Configuration Example #########################

This file is an example configuration file highlighting only the most common

options. The filebeat.reference.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

gsuboc · February 4, 2020, 5:16pm

I’ve commented the multiline for filebeat and here’s the result. It still shows under message.

inhinyera16 · February 4, 2020, 7:47pm

You need to show us ur filebeat file, bcoz by default, filebeat will read your file line per line unless you have enabled multiline pattern. I can see a multiline config in ur screenshot. You need to disable it. Then use my filter above. If you can’t provide us what we need then there’s no way we can help you.

gsuboc · February 4, 2020, 8:01pm

Attached is the current filebeat file with the filter you have provided. Please let me know if you need other info.

Thanks.

gsuboc · February 5, 2020, 9:30pm

Does this screenshot helps you out?

inhinyera16 · February 6, 2020, 1:15pm

Yes, disable multiline. Its normal that it's under message field. Now, use grok filter to parse that out from message field.

grok { 
		break_on_match => true
		match => { "message" => [ 
			"Job_Name: %{WORD:Job_Name}",
			"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
			"FrameCount: %{NUMBER:FrameCount}",
			"FrameList: %{NUMBER:FrameList}",
			"Render_Type: %{WORD:Render_Type}",
			"UserSubmitter: %{WORD:UserSubmitter}",
			"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",
			"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",
			"SubmitDate: %{DATESTAMP:SubmitDate}",
			"CompletedDate: %{DATESTAMP:CompletedDate}",
			"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",
			"JobPriority: %{NUMBER:JobPriority}",
			"Job_ID: %{WORD:Job_ID}" ] }
		}

gsuboc · February 6, 2020, 2:43pm

Yes, that’s what I have on my logstash but I can’t aggregate and visualize that field.

gsuboc · February 6, 2020, 10:55pm

Is there a way to make my fields aggregatable?

gsuboc · February 10, 2020, 9:57pm

Do you have a recommendation? Thanks.