Need grok pattern for logstash

Elastic Stack Logstash
1

Hi All,

I need to use grok filter to parse the following log pattern:

[2022-05-24T02:15:20.979+0000][info][gc             ] GC(187) Pause Full (G1 Evacuation Pause) 2559M->1698M(2560M) 724.899ms

The idea is to capture the following from the log:

  1. Date/Time

  2. 2559M->1698M showing drop in the heap due to garbage collection

  3. 2560M Total heap

  4. 724.899ms Time take for GC

I get stumped at the first pattern for date/ timestamp and following does not seem to work when I put in the debugger:

%{TIMESTAMP_ISO8601:time}

Please guide what filters should I use for the entire log message?

Thanks

1 Like
2

Hello @zaeemmasood

Hope this grok pattern would help you

\[%{TIMESTAMP_ISO8601:date_time}\]\[%{WORD:loglevel}\]\[%{WORD:gc}\s*\]\s+%{GREEDYDATA:message} %{GREEDYDATA:drop_heap}\(%{DATA:total_gc}\) %{GREEDYDATA:time_taken}

Attached screenshot for your reference.

grok2
grok2788×1534 22.1 KB

Keep Posted!!! Thanks :slight_smile: !!!

3

Thanks. That solves my issue. I need to display the fields individually in Kibana. This does not seem to work:

        filter {
                if [type] == "tv_gclog_analysis"  {

                        grok {
                                match => { "message" => "\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}" }
                                        }

                               }
                        }

Do I need to somehow split the message as follows:

        if [type] == "tv_gclog_analysis"  {
                
				   filter {
					grok {
						match => { "message" => \[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int} }
					}
					
				mutate {
                        split => ["message",  ]
                        add_field =>{
                           "createdTime" => "%{[message][0]}"
                           "logLevel" => "%{[message][1]}"
                           "message" => "%{[message][2]}"
                           "heapDrop" => "%{[message][3]}"
			               "maxHeap" => "%{[message][4]}"
			               "time_taken" => "%{[message][5]}"

                       }
					}
				}

Please guide

4

Hello @zaeemmasood

this grok pattern should be inside single quotes like the below . if still not working please a raise a new case to assist you further.

 filter {
					grok {
						match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}' }
					}
5

Thanks for your help!

Please let me know if there is a central place where all the possible grok patterns are mentioned?

6

Hello @zaeemmasood

In this below elastic article you would find a git repo where it contains mostly all the patterns.

7

Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.