Logstash Grok pattern not working

Elastic Stack Logstash
1

hi There ,

I am completely new to ELK and currently learning to configure it for different logs,

ELK version 7.0

while parsing the GClogs to logstash through fileBeat I see the pattrent match fails( please note same pattern works fine with grok debugger)
sample GC log:-

2019-05-15T14:24:56.355+0100: 0.171: [GC (Allocation Failure) [PSYoungGen: 2048K->480K(2560K)] 2048K->1032K(9728K), 0.0033195 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]

till 2019-05-15T14:24:56.355+0100: 0.171: [GC (Allocation Failure) [PSYoungGen: 2048K am able to to parse but not"->480K"**

> input {
>   beats {
>     port => 5044
>   }
> }
> 
> filter {
>     grok {
>        match => {"message" => "%%{TIMESTAMP_ISO8601:timestamp}: %{NUMBER:jvm_time}: \[%{DATA:gc_type} \(%{DATA:gc_cause}\) \[%{DATA:TYPE}: %{NUMBER:beforeGC}\K\-\>%{NUMBER:AfterGC}\K\(%{NUMBER:young_generation_total}\K\)\] %{NUMBER:HeapbeforeGC}\K\-\>%{NUMBER:HeapAfterGC}\K\(%{NUMBER:totalHeap}\K\)\, %{NUMBER:totaltime}.*?\] \[%{DATA}: %{DATA}=%{NUMBER:user_time} %{DATA}=%{NUMBER:sys}, %{DATA}=%{NUMBER:totalsec}"}
>     tag_on_failure => ["tags", "gcinspector_grok_parse_failure"]
>       }
> }
> 
>   output {
>   elasticsearch {
>                         hosts => "localhost:9200"
>                 }
> }

can you please help me to know what mistake i m making here

2

You need to edit your post and format it so that it is readable. Select the configuration and click on </> in the toolbar above the edit pane. Check it is readable in the preview pane on the right. It is there for that reason.

3

Sorry :frowning: my bad though I used the </> ,due file config file name ..it was not formatted,its done now ,can you please check now

4

You have an extra % at the start of the line which should probably be ^

You do not need to escape K or > with backslash.

That said, I think you should capture the K along with the number, because your code is going to break when it sees B or M or G there. If you capture 9728K using something like (?<someSize>[0-9]+[BKMGT]).

You then need to convert that to a number. I once did that will a mutate filter (replacing K with 000 etc -- good enough for what I needed right then). I thought I recently saw a filter that could do that, but I cannot find it now. It may have been an elasticsearch mapper, but I cannot find that either. This thread has some suggestions around that, including ruby code.

5

Thanks much ..was able to parse the GC logs and can split the details as required ,please find the below sample output of "Discover" window

    Time  	                    TYPE   	gc_type gc_cause     	   totalHeap UsedAafterGC   before_gc   real sys       	YoungGenafter_gc  youngUsedAftergc       	youngGenbeforeGC     
	May 20, 2019 @ 15:02:05.701	PSYoungGen	GC	Allocation Failure	28,160	10,913	         15,577    	0	0	          9,216          	1,238	                 5,910

now I need to plot a graph for youngen and oldgen Heap pattern along with GC cause and GC duration ...could you please let me know how i can do tht ( i tried with line and Data table but not able figure out

Thanks again
Bhaskar

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.