Grok filter for logstash/filebeats

Elastic Stack Logstash
1

I'm trying to extract my logs field using grok but it doesn't work.

Here's my grok filter:

filter { # parse the CSV structure generated from the log file into fields grok { match => {"message" => "%{GREEDYDATA:Job_Name} %{HAPROXYTIME:ElapsedRenderTime} %{NUMBER:FrameCount} %{WORD:FrameList} %{WORD:Render_Type} %{WORD:UserSubmitter} %{HAPROXYTIME:TotalTaskRenderTime} %{HAPROXYTIME:JobAverageFrameRenderTime} %{DATESTAMP:SubmitDate} %{DATESTAMP:CompletedDate} %{WORD:MachineSubmitted} %{NUMBER:JobPriority} %{WORD:Job_ID}" } } }

Here's my logfile:

Job_Name: TESTj
ElapsedRenderTime: 00:00:27.8370000
FrameCount: 1
FrameList: 21
Render_Type: AfterEffects
UserSubmitter: user1
TotalTaskRenderTime: 00:00:23.2210000
JobAverageFrameRenderTime: 00d:00h:00m:23s
SubmitDate: 01/24/2020 09:56:17
CompletedDate: 01/24/2020 09:56:47
MachineSubmitted: D1-RN-XX01
JobPriority: 50
Job_ID: 5e2b2fc1dddcae24084784b7

I've also tried %{WORD:Job_Name} but the result is this:

image
image604×1093 36.9 KB

The result is incorrect. This should have been "TESTj" instead of "Job_Name".

2

I would have grok match that against a list of patterns. If you are using a multiline filter to combine the logfile into a single event you will need 'break_on_match => false'

grok {
    match => {
        "message" => [
            "Job_Name: %{WORD:Job_Name}",
            "ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
            "FrameCount: %{NUMBER:FrameCount}",
            ...
        ]
    }
}

I am not convinced that 00:00:27.8370000 matches the HAPROXYTIME pattern.

3

I would recommend trying kv filter instead of grok.

4

Does "break_on_match" would go under file or filter section? I have used the grok debugger online and HAPROXYTIME is what it showed for ElapsedRenderTime filed.

5

How would I use kv filter? thanks.

6

You would add

break_on_match => false

to the grok filter.

7

does it require a pattern directory?

8

No, it does not.

9

filter {
# parse the CSV structure generated from the log file into fields

break_on_match => false
grok { match => { "message" => [ "Job_Name: %{WORD:Job_Name}",
								"ElapsedRenderTime: %HAPROXYTIME:ElapsedRenderTime}", 
								"FrameCount: %{NUMBER:FrameCount}",
								"FrameList: %{WORD:FrameList}",
								"Render_Type: %{WORD:Render_Type}",
								"UserSubmitter: %{WORD:UserSubmitter}",
								"TotalRenderTime: %{HAPROXYTIME:TotalRenderTime}",
								"JobAverageFrameRenderTime: %{HAPROXYTIME:JobAverageFrameRenderTime}",
								"SubmitDate: %{TIMESTAMP:SubmitDate}",
								"CompletedDate: %{TIMESTAMP:CompletedDate}",
								"MachineSubmitted: %{WORD:MachineSubmitted}",
								"JobPriority: %{NUMBER:JobPriority}",
								"Job_ID: %{WORD:Job_ID}" ] }

	}

Here's my grok filter but running the logstash service shows successful but it shutdown. I'm also running filebeat but logstash.yml is disabled.
I still don't see the result my looking for.

10

The break_on_match option has to be inside the grok filter.

11

Here's my complete config:

input {
file {
path => "\d1motion-fs\kibana_logs\render_logs*.txt"
start_position => "beginning"
}
}

filter {
grok { break_on_match => false
match => { "message" => [ "Job_Name: %{WORD:Job_Name}",
"ElapsedRenderTime: %{HAPROXYTIME:ElapsedRenderTime}",
"FrameCount: %{NUMBER:FrameCount}",
"FrameList: %{WORD:FrameList}",
"Render_Type: %{WORD:Render_Type}",
"UserSubmitter: %{WORD:UserSubmitter}",
"TotalRenderTime: %{HAPROXYTIME:TotalRenderTime}",
"JobAverageFrameRenderTime: %{HAPROXYTIME:JobAverageFrameRenderTime}",
"SubmitDate: %{TIMESTAMP:SubmitDate}",
"CompletedDate: %{TIMESTAMP:CompletedDate}",
"MachineSubmitted: %{WORD:MachineSubmitted}",
"JobPriority: %{NUMBER:JobPriority}",
"Job_ID: %{WORD:Job_ID}" ] }

	}

}
output {

elasticsearch {
	hosts => ["D1motion-fs:9200"]
	
}

}

12

Please let me know if there's any error on this config. I'm still learning this thing.

13

Do not use backslash in the path option of a file input. Use forward slash.

14

It's a windows platform. I have tried the // but it doesn't grab my files from that location.

15

It's actually this:

path => "\\d1motion-fs\kibana_logs\render_logs\*.txt"

16

As I said, do not use backslash. Try

path => "/d1motion-fs/kibana_logs/render_logs/*.txt"
17

Changed path to:
path => "//d1motion-fs/kibana_logs/render_logs/*.txt"

I'll run some test and will post the result, thanks.

18

Here's my result:

log.flags multiline
log.offset 0
message Job_Name: TESTq
ElapsedRenderTime: 00:00:28.8190000
FrameCount: 1
FrameList: 8
Render_Type: AfterEffects
UserSubmitter: gssuboc
TotalTaskRenderTime: 00:00:24.3090000
JobAverageFrameRenderTime: 00d:00h:00m:24s
SubmitDate: 01/27/2020 10:34:42
CompletedDate: 01/27/2020 10:35:12
MachineSubmitted: D1-RN-XX01
JobPriority: 50

The fields are still under the "message" section, not separated.

19

What do you get when you use

output { stdout { codec => rubydebug } }

?

20

same result as above.