Logstash-Not able to grok parse the incoming message field properly with condition

Elastic Stack Logstash
1

Hello,

Below is my incoming single log message:

[2024-02-14T03:29:07.962+0000][118][safepoint   ] Safepoint "Cleanup", Time since last: 1000174402 ns, Reaching safepoint: 9291 ns, Cleanup: 83971 ns, At safepoint: 14116 ns, Total: 107378 ns
[2024-02-14T03:29:08.373+0000][118][gc,start    ] GC(16) Pause Young (Concurrent Start) (Metadata GC Threshold)
[2024-02-14T03:29:08.373+0000][118][gc,task     ] GC(16) Using 4 workers of 4 for evacuation
[2024-02-14T03:29:08.373+0000][118][gc,age      ] GC(16) Desired survivor size 40894464 bytes, new threshold 15 (max threshold 15)
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) Age table with threshold 15 (max threshold 15)
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   1:   13258944 bytes,   13258944 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   2:    1069200 bytes,   14328144 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   3:     754040 bytes,   15082184 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   4:     118080 bytes,   15200264 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   5:     322488 bytes,   15522752 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   6:     262400 bytes,   15785152 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   7:     431192 bytes,   16216344 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   8:     269312 bytes,   16485656 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age   9:    3006824 bytes,   19492480 total
[2024-02-14T03:29:08.387+0000][118][gc,age      ] GC(16) - age  10:     177680 bytes,   19670160 total
[2024-02-14T03:29:08.387+0000][118][gc,phases   ] GC(16)   Pre Evacuate Collection Set: 0.2ms
[2024-02-14T03:29:08.387+0000][118][gc,phases   ] GC(16)   Merge Heap Roots: 0.2ms
[2024-02-14T03:29:08.387+0000][118][gc,phases   ] GC(16)   Evacuate Collection Set: 12.5ms
[2024-02-14T03:29:08.387+0000][118][gc,phases   ] GC(16)   Post Evacuate Collection Set: 1.2ms
[2024-02-14T03:29:08.387+0000][118][gc,phases   ] GC(16)   Other: 0.4ms
[2024-02-14T03:29:08.387+0000][118][gc,heap     ] GC(16) Eden regions: 180->0(297)
[2024-02-14T03:29:08.387+0000][118][gc,heap     ] GC(16) Survivor regions: 4->10(39)
[2024-02-14T03:29:08.387+0000][118][gc,heap     ] GC(16) Old regions: 20->20
[2024-02-14T03:29:08.387+0000][118][gc,heap     ] GC(16) Archive regions: 2->2
[2024-02-14T03:29:08.387+0000][118][gc,heap     ] GC(16) Humongous regions: 3->3
[2024-02-14T03:29:08.387+0000][118][gc,metaspace] GC(16) Metaspace: 36172K(36608K)->36172K(36608K) NonClass: 32501K(32704K)->32501K(32704K) Class: 3671K(3904K)->3671K(3904K)
[2024-02-14T03:29:08.387+0000][118][gc          ] GC(16) Pause Young (Concurrent Start) (Metadata GC Threshold) 413M->67M(1024M) 14.537ms
[2024-02-14T03:29:08.387+0000][118][gc,cpu      ] GC(16) User=0.05s Sys=0.00s Real=0.02s

I have copied message to another field and able to split the message and get independent lines as separate messages with below:

mutate {
		copy => {
			"message" => "message_copy"
		}
	}
	
	split {field => "message_copy"}

After the split which is working, i need to search a string [Eden] in incoming message_copy field and then do some action, using below


> if [Eden] in [message_copy]
> 		{	
> 				mutate {
> 				copy => {
> 				"message_copy" => "eden_copy"
> 				}
> 			}
> 	 	
> 		   grok 
> 			{
> 			match => ["eden_copy", "(\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{NUMBER:id}\]\[%{NOTSPACE:gcinfo}%{SPACE}\] %{NOTSPACE}\(%{NUMBER:gcnum}\) %{NOTSPACE} %{NOTSPACE} %{NOTSPACE:edenbeforegc}\-\>%{NOTSPACE:edenaftergc}\(%{NOTSPACE:totaleden}\))"]
> 			}
> 		}

Not able to get the fields out. Pls let me know the issue

2

That tests whether the value of the field [Eden] can be found in the field [message_copy]. You should change it to if "Eden" in [message_copy], then it works

   "totaleden" => "297",
"edenbeforegc" => "180",
      "gcinfo" => "gc,heap",
          "id" => "118",
       "gcnum" => "16",
 "edenaftergc" => "0",
3

First i have tried that only and it didn't work and then i changed to If [Eden]..Able to get all separate lines as different message _copy lines, but "Eden.." message _copy line is not even coming. Pls check

pls find the complete filter below:

filter {
	if [fields][hc_type] and [fields][hc_type] == "gc-log" 
	{
		mutate {
			copy => {
				"message" => "message_copy"
			}
		}
	
		split {field => "message_copy"}
	
		if "Eden" in [message_copy]
		{	
			mutate {
				copy => {
				"message_copy" => "eden_copy"
				}
			}
	
		   grok 
			{
			match => ["eden_copy", "(\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{NUMBER:id}\]\[%{NOTSPACE:gcinfo}%{SPACE}\] %{NOTSPACE}\(%{NUMBER:gcnum}\) %{NOTSPACE} %{NOTSPACE} %{NOTSPACE:edenbeforegc}\-\>%{NOTSPACE:edenaftergc}\(%{NOTSPACE:totaleden}\))"]
			}

		}		
	}	
}
4

Working fine now..Thanks