Grok filter for logstash/filebeats

Elastic Stack Logstash
21

any recommendation?

22

You didn't answer my question, so I cannot add anything.

23

I'm getting the same result with output { stdout { codec => rubydebug } }.

24

I am asking you to show me what that rubydebug codec produces.

25

Here’s the result:

image001.png
image001.png1041×673 55.3 KB

26

Ok, using this log:

Job_Name: TESTj
ElapsedRenderTime: 00:00:27.8370000
FrameCount: 1
FrameList: 21
Render_Type: AfterEffects
UserSubmitter: user1
TotalTaskRenderTime: 00:00:23.2210000
JobAverageFrameRenderTime: 00d:00h:00m:23s
SubmitDate: 01/24/2020 09:56:17
CompletedDate: 01/24/2020 09:56:47
MachineSubmitted: D1-RN-XX01
JobPriority: 50
Job_ID: 5e2b2fc1dddcae24084784b7

and this pipeline:

input {
	file {
		path => "/tmpBIS/ror/elastic_env/logstash-6.7.1/testlog"
		start_position => "beginning"
 		sincedb_path => "/dev/null"
	}
}


filter {
	grok { 
		break_on_match => true
		match => { "message" => [ 
			"Job_Name: %{WORD:Job_Name}",
			"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
			"FrameCount: %{NUMBER:FrameCount}",
			"FrameList: %{NUMBER:FrameList}",
			"Render_Type: %{WORD:Render_Type}",
			"UserSubmitter: %{WORD:UserSubmitter}",
			"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",
			"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",
			"SubmitDate: %{DATESTAMP:SubmitDate}",
			"CompletedDate: %{DATESTAMP:CompletedDate}",
			"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",
			"JobPriority: %{NUMBER:JobPriority}",
			"Job_ID: %{WORD:Job_ID}" ] }
		}

	mutate {
		remove_field => [ "message","host","@version","path","@timestamp","haproxy_minute","haproxy_hour","haproxy_second" ]
	}
}
output {
	stdout {codec=> rubydebug}
}

Will result to :

{
    "FrameList" => "21"
}
{
    "JobAverageFrameRenderTime" => "00d:00h:00m:23s"
}
{
    "JobPriority" => "50"
}
{
    "FrameCount" => "1"
}
{
    "TotalTaskRenderTime" => "00:00:23.2210000"
}
{
    "MachineSubmitted" => "D1-RN-XX01"
}
{
    "Job_Name" => "TESTj"
}
{
    "Render_Type" => "AfterEffects"
}
{
    "SubmitDate" => "01/24/2020 09:56:17"
}
{
    "ElapsedRenderTime" => "00:00:27.8370000"
}
{
    "UserSubmitter" => "user1"
}
{
    "CompletedDate" => "01/24/2020 09:56:47"
}
27

Thanks for replying. Here’s the latest result with your current config.

image001.png
image001.png810×500 40.4 KB

28

Here’s the config file:

input {

file {

path => "\D1motion-fs\kibana_logs\render_logs\render*.log"

start_position => "beginning"

sincedb_path => "/dev/null"

}

}

filter {

grok { break_on_match => false

match => { "message" => [

"Job_Name: %{WORD:Job_Name}",

"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",

"FrameCount: %{NUMBER:FrameCount}",

"FrameList: %{NUMBER:FrameList}",

"Render_Type: %{WORD:Render_Type}",

"UserSubmitter: %{WORD:UserSubmitter}",

"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",

"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",

"SubmitDate: %{DATESTAMP:SubmitDate}",

"CompletedDate: %{DATESTAMP:CompletedDate}",

"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",

"JobPriority: %{NUMBER:JobPriority}",

"Job_ID: %{WORD:Job_ID}" ] }

}

mutate {

remove_field => [ "message","host","@version","path","@timestamp","haproxy_minute","haproxy_hour","haproxy_second" ]

}

}

output {

stdout { codec => rubydebug }

}

image001.png
image001.png810×500 40.4 KB

29

Still not separating the fieldnames from the log.

30

Any other recommendation? Thanks.

31

can you post your filebeat config? looks like your filebeat is reading your logs using multiline patterns

32

Here you go.

###################### Filebeat Configuration Example #########################

This file is an example configuration file highlighting only the most common

options. The filebeat.reference.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

33

I’ve commented the multiline for filebeat and here’s the result. It still shows under message.

image001.png
image001.png741×205 10.7 KB

image002.png
image002.png1364×459 73.6 KB

34

You need to show us ur filebeat file, bcoz by default, filebeat will read your file line per line unless you have enabled multiline pattern. I can see a multiline config in ur screenshot. You need to disable it. Then use my filter above. If you can’t provide us what we need then there’s no way we can help you.

35

Attached is the current filebeat file with the filter you have provided. Please let me know if you need other info.

Thanks.

image
image1518×1352 170 KB
image
image1538×1177 67.8 KB
image
image1660×1306 176 KB
image
image1398×1241 52.6 KB
image
image1480×499 27.7 KB

37

Does this screenshot helps you out?

38

Yes, disable multiline. Its normal that it's under message field. Now, use grok filter to parse that out from message field.

grok { 
		break_on_match => true
		match => { "message" => [ 
			"Job_Name: %{WORD:Job_Name}",
			"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
			"FrameCount: %{NUMBER:FrameCount}",
			"FrameList: %{NUMBER:FrameList}",
			"Render_Type: %{WORD:Render_Type}",
			"UserSubmitter: %{WORD:UserSubmitter}",
			"TotalTaskRenderTime: %{HAPROXYTIME:TotalTaskRenderTime}",
			"JobAverageFrameRenderTime: %{GREEDYDATA:JobAverageFrameRenderTime}",
			"SubmitDate: %{DATESTAMP:SubmitDate}",
			"CompletedDate: %{DATESTAMP:CompletedDate}",
			"MachineSubmitted: %{GREEDYDATA:MachineSubmitted}",
			"JobPriority: %{NUMBER:JobPriority}",
			"Job_ID: %{WORD:Job_ID}" ] }
		}
39

Yes, that’s what I have on my logstash but I can’t aggregate and visualize that field.

40

Is there a way to make my fields aggregatable?

41

Do you have a recommendation? Thanks.