Filter for logstash

Elastic Stack Logstash
1

launch TIMESTAMP START: 2016.11.15 05:30:23
INFO: 200000 packets. Current packet is class com.navtech.kernel.flat.FlatCombo [Loader] [tstamp: 1866 time: 1851.4 (30:51) split: 03.2] [Memory: 3.320G total: 22.20G free: 18.88G]

INFO: [BatchValidationCommand] [tstamp: 4992 time: 39.1 split: 00.0][Memory:18.95G total: 37.49G free: 18.54G]

INFO: Done executing all batch validations. [ValidationService] [tstamp: 4992 time: 38.7 split: 38.7] [Memory: 18.95G total: 37.49G free: 18.54G][save rule violations] [tstamp: 4998 time: 4997.6 (1:23:17) split: 4997.6 (1:23:17)] [Memory:23.25G total: 37.49G free: 14.24G]

launch TIMESTAMP END: 2016.11.15 06:53:45

what will be the filter for this log, if I want to extract data like timestamp, time consumed per process and memory?

Thanks

2

What have you tried?

3

INFO: 200000 packets. Current packet is class com.navtech.kernel.flat.FlatCombo [Loader] [tstamp: 1866 time: 1851.4 (30:51) split: 03.2] [Memory: 3.320G total: 22.20G free: 18.88G]

for this, I tried
\A(?:%{LOGLEVEL:level})%{GREEDYDATA:message}\s+([%{WORD:loglevel}]\s+)+([%{GREEDYDATA:ts}]\s+)+([%{GREEDYDATA:mem}])

but now what I'm missing is to extract the "Memory:" and "total:" so how will i write regex for that

4

Having more than one GREEDYDATA pattern in an expression can be problematic as well as inefficient. What are expecting the parsed event to look like?

5

Hi, I'm expecting it to look like

log level
message
class
memory used/memory free
and timestamp

further I enhanced it to
\A(?:%{LOGLEVEL:level})%{GREEDYDATA:message}\s(?:%{JAVACLASS:caller_class})\s+([%{WORD:log}]\s+)+([%{DATA:ts}]\s+)+([%{DATA:mem}])

6

What exactly do you expect to capture in those fields? Can you create an expected JSON document based on the example event?

8

I was using the grok debugger to verify my regex pattern, I want the data tokenized so that I can visualize it in Kibana like process vs time or time vs memory graph

9

Can you please create an example JSON document that show how you would like the example event parsed and post it here? This will make it a lot easier for us to help you.

10

{
"level": [
"INFO"
],
"message": [
"200000 packets. Current packet is class"
],
"caller_class": [
"com.navtech.kernel.flat.FlatCombo"
],
"loglevel": [
"Loader"
],
"tstamp": [
"1866"
],
"memory": [
"3.320"
],
"total": [
"22.20"
]
}

This was the expectation from my command

11

^(?:%{LOGLEVEL:level}):\s*%{DATA:message}\s*(?:%{JAVACLASS:caller_class})\s+[%{WORD:loglevel}]\s+([\s*\S+\s+%{BASE10NUM:tstamp}.?]\s+)+([\s\S+\s+%{BASE10NUM:memory}\S*\s+\S+\s+%{BASE10NUM:total}.*?])

I got the result fine with this from stackoverflow, but how shall I use this grok in the filter command in logstash conf

12

Use grok to separate out the various parts into separate fields. You should then e.g. be able to use the kv filter to parse out memory and total from the last part, although you may use a mutate filter to clean the fields up and remove the 'G' training the number.

13

I'm sorry I didn't understand what you meant to say

how shall I use grok to separate it into fields, I used http://grokdebug.herokuapp.com/ to match my pattern and the particular pattern matched, but how shall I use this Info

can you provide some URLs or wiki links or videos for reference

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.