Assigning an aggregated event timestamp to @timestamp

Elastic Stack Logstash
1

HI,

I am aggragating events for each session from log
I fetch the log time from event

in aggregate filter, i want to have a field startdatetime and assign session open time to this,
set timeout_timestamp_field => "StartDateTime",
Then assign this "StartDateTime" to @timestamp

first part works fine but i get date parse error . What am i missing ?
Snippet from Filter plugin

mutate {
add_field => {
"StartDateTime" => "%{log_time}"
}
}

date {
match => [ "log_time" , "MMM dd HH:mm:ss" ]
timezone => "EST5EDT"
target => "log_time"
}

date {
match => [ "StartDateTime" , "MMM dd HH:mm:ss" ]
timezone => "EST5EDT"
target => "StartDateTime"
}

if ([msg] =~ /^session opened /) {

 aggregate {
   task_id => "%{sessionId}"
   code => "
            map['StartDateTime'] = event.get('log_time')          
            
            event.cancel()"
   map_action => "create"
   
 }

}
more aggregate filters

if ([msg] =~ /^session closed /) {

 aggregate {
   task_id => "%{sessionId}"
   code => "
            event.set('StartDateTime', map['StartDateTime'])"         
            
    map_action => "update"
   end_of_task => true
   timeout => 6
   timeout_timestamp_field => "StartDateTime"
   timeout_task_id_field => "sessionId"       
   push_map_as_event_on_timeout => true       
   
 }

}

2

Presumably the date format does not match the pattern you are using, but you have not shown us what your log_time field looks like.

3

sample log_time : Apr 9 15:15:11
log_time and StartDateTime gets properly populated in the output
Output:
"StartDateTime" => 2020-04-09T19:14:46.000Z
"log_time" => 2020-04-09T19:15:11.000Z

Sorry missed to add another filter which i added after all aggregate filters, that is giving me the error i guess

if[StartDateTime]
{
date {
match => [ "StartDateTime" , "YYYY-MM-dd'T'HH:mm:ss.SSSZ", "MMM dd HH:mm:ss", "MMM d HH:mm:ss" , "ISO8601"]
timezone => "EST5EDT"

   }   
 }
4

If that is rubydebug output then both are already of type Logstash::Timestamp, which is what a date filter creates. A date filter cannot parse that.

5

how do i assign StartDateTime to @timestamp ?
I tried below in last aggregate filter and in timeout code, it sets the value of @timestamp as StartDateTime for normal events but gives aggregateException on timeout events and doesn't set @timestamp with the value of StartDateTime for timeout events
event.set('@timestamp', map['StartDateTime'])
timeout_code => "
event.set('@timestamp', map['StartDateTime'])"

what is the right way to do so ?

6

Appreciate any help

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.