Logstash Grok pattern not working

hi There ,

I am completely new to ELK and currently learning to configure it for different logs,

ELK version 7.0

while parsing the GClogs to logstash through fileBeat I see the pattrent match fails( please note same pattern works fine with grok debugger)
sample GC log:-

2019-05-15T14:24:56.355+0100: 0.171: [GC (Allocation Failure) [PSYoungGen: 2048K->480K(2560K)] 2048K->1032K(9728K), 0.0033195 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]

till 2019-05-15T14:24:56.355+0100: 0.171: [GC (Allocation Failure) [PSYoungGen: 2048K am able to to parse but not"->480K"**

> input {
>   beats {
>     port => 5044
>   }
> }
> 
> filter {
>     grok {
>        match => {"message" => "%%{TIMESTAMP_ISO8601:timestamp}: %{NUMBER:jvm_time}: \[%{DATA:gc_type} \(%{DATA:gc_cause}\) \[%{DATA:TYPE}: %{NUMBER:beforeGC}\K\-\>%{NUMBER:AfterGC}\K\(%{NUMBER:young_generation_total}\K\)\] %{NUMBER:HeapbeforeGC}\K\-\>%{NUMBER:HeapAfterGC}\K\(%{NUMBER:totalHeap}\K\)\, %{NUMBER:totaltime}.*?\] \[%{DATA}: %{DATA}=%{NUMBER:user_time} %{DATA}=%{NUMBER:sys}, %{DATA}=%{NUMBER:totalsec}"}
>     tag_on_failure => ["tags", "gcinspector_grok_parse_failure"]
>       }
> }
> 
>   output {
>   elasticsearch {
>                         hosts => "localhost:9200"
>                 }
> }

can you please help me to know what mistake i m making here

You need to edit your post and format it so that it is readable. Select the configuration and click on </> in the toolbar above the edit pane. Check it is readable in the preview pane on the right. It is there for that reason.

Sorry :frowning: my bad though I used the </> ,due file config file name ..it was not formatted,its done now ,can you please check now

You have an extra % at the start of the line which should probably be ^

You do not need to escape K or > with backslash.

That said, I think you should capture the K along with the number, because your code is going to break when it sees B or M or G there. If you capture 9728K using something like (?<someSize>[0-9]+[BKMGT]).

You then need to convert that to a number. I once did that will a mutate filter (replacing K with 000 etc -- good enough for what I needed right then). I thought I recently saw a filter that could do that, but I cannot find it now. It may have been an elasticsearch mapper, but I cannot find that either. This thread has some suggestions around that, including ruby code.

Thanks much ..was able to parse the GC logs and can split the details as required ,please find the below sample output of "Discover" window

    Time  	                    TYPE   	gc_type gc_cause     	   totalHeap UsedAafterGC   before_gc   real sys       	YoungGenafter_gc  youngUsedAftergc       	youngGenbeforeGC     
	May 20, 2019 @ 15:02:05.701	PSYoungGen	GC	Allocation Failure	28,160	10,913	         15,577    	0	0	          9,216          	1,238	                 5,910

now I need to plot a graph for youngen and oldgen Heap pattern along with GC cause and GC duration ...could you please let me know how i can do tht ( i tried with line and Data table but not able figure out

Thanks again
Bhaskar

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.