Aggregate filter not working correctly iwth logstash

Elastic Stack Logstash
1

Here is my input log file names as : cadence-cdslmd-dlhl0939_dlhl0940_dlhl0941-5.11.2019-event.log
15:56:24 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
15:56:32 (cdslmd) OUT: "Analog_Design_Environment_XL" gautamd@dlhsx00005
15:56:33 (lmgrd) TIMESTAMP 5/13/2019
15:56:35 (cdslmd) OUT: "OASIS_Simulation_Interface" mnarasim@dlhsx00012
15:56:37 (cdslmd) IN: "111" guptasri@dlhsx00010
15:56:42 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012

My logstash filter
input {
beats {
port => 5044
}
}

filter {
		
grok {
	match => [ "message", "%{DATA:event_timestamp} \(%{DATA:lmgrd}\) TIMESTAMP %{DATE:monthday}" ]
	tag_on_failure => [ "message_data" ]
		}
mutate {
	add_field => { "taskId" => "all" }
	}

if "message_data" not in [tags] {
	aggregate {
		task_id => "%{taskId}"
		code => "map['monthday'] = event.get('monthday')"
				}
}
else {
		aggregate {
		task_id => "%{taskId}"
		code => "if (map['monthday'] != nil) then event.set('monthday', map['monthday']) else event.set('monthday', (event.get('source').split('-')[3])) end"
		}

	grok { 
		 match => [ "message", "%{TIME:event_timestamp} \(%{DATA:lic_vendor_name}\) (?<event_type>(OUT|IN|DENIED))\: \"%{DATA:lic_feature_name}\" %{DATA:user_name}@%{HOSTNAME:host_name}" ] }
	
	mutate { replace => ["event_timestamp", "%{monthday} %{event_timestamp}" ] }
	
	date { match => [ "event_timestamp", "M/d/yyyy HH:mm:ss","M.d.yyyy HH:mm:ss","ISO8601" ] target => "event_timestamp" }
}

}
output {
stdout {
codec => rubydebug
}
}

Before input line 3, the monthday should come from filename( 3rd field :5.11.2019) and after that monthday should get from the logfile itself from TIMESTAMP line ( 5/13/2019)

The output coming corresponding to input lines are
Line 1 : "event_timestamp" => 2019-05-11T10:26:24.000Z, "monthday" => "5.11.2019"
Line 2: "event_timestamp" => 2019-05-11T10:26:32.000Z "monthday" => "5.11.2019"
Line3 : "event_timestamp" => "15:56:33" "monthday" => "5/13/2019"
Line4 : "event_timestamp" => 2019-05-13T10:26:35.000Z "monthday" => "5/13/2019"
Line 5: "event_timestamp" => 2019-05-11T10:26:37.000Z "monthday" => "5.11.2019"
Line 6: "event_timestamp" => 2019-05-11T10:26:42.000Z, "monthday" => "5.11.2019"

Expected output should be
Line 1 : "event_timestamp" => 2019-05-11T10:26:24.000Z, "monthday" => "5.11.2019"
Line 2: "event_timestamp" => 2019-05-11T10:26:32.000Z "monthday" => "5.11.2019"
Line3 : "event_timestamp" => "15:56:33" "monthday" => "5/13/2019"
Line4 : "event_timestamp" => 2019-05-13T10:26:35.000Z "monthday" => "5/13/2019"
Line 5: "event_timestamp" => 2019-05-11T10:26:37.000Z "monthday" => "5/13/2019"
Line 6: "event_timestamp" => 2019-05-11T10:26:42.000Z, "monthday" => "5/13/2019"

Please suggest where I am doing it wrong?

2

Does setting '--pipeline.batch.size 1' help?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.