Grok fiels are removed by aggregate section

I have follwoing file structure:

id, iduser,datetimInit, dateTimeends
0001 0001 2023-02-03 04:45:16.78 2023-02-03 04:46:16.78
0002 0001 2023-02-03 08:45:16.78 2023-02-03 08:46:16.78
0003 0002 2023-02-04 04:45:16.78 2023-02-04 04:46:16.78

Where: I need to add duration time (datetimeEnds - dateTimeInit) , and full duration (sum duration by idUser) time by iduser. i have the following configuration file:

filter {	
	grok {
		match => { 
			"message" => "%{NUMBER:id} %{NUMBER:idUser} %{TIMESTAMP_ISO8601:dateTimeInit} %{TIMESTAMP_ISO8601:dateTimeEnds}" 
		}
	}
	date { match => ["dateTimeInit", "ISO8601"] target => "dateTimeInit" }
	date { match => ["dateTimeEnds", "ISO8601"] target => "dateTimeEnds" }
	
	#add duration by row
	ruby { 
		code => "event.set('duration', event.get('dateTimeEnds').to_i - event.get('dateTimeInit').to_i); "
	}
	
	#add fullduration depending on same iduser
	aggregate {
       task_id => "%{idUser}"
       code => "
			map['full_duration'] ||= 0 ; 
			map['full_duration'] += event.get('duration')
			event.cancel()
		"
	   push_previous_map_as_event => true
       timeout => 3
    }
}

This is the output:

{
    "full_duration" => 2,
       "@timestamp" => 2023-02-06T13:08:18.923849100Z,
         "@version" => "1"
}

Where Dates and idUser fields are missing.

How could I add all the fields to the output, not just full_duration?

Thank you in advance

That pushes the previous map as an event. The only field you added to the map is map['full_duration'], so that is the only non-standard field that your event has. You need to copy the other fields that you want to preserve to fields in the map hash.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.