Calculate elapsed time for multiple end events

[Patrick_Gell]

Hello,

I have a message driven system where one message is sent to multiple receiver.
Now I wanted to track the time it takes for every receiver until the message arrives.
The structure of my log entries looks like:
TIMESTAMP SYSTEMID MESSAGEID
20190101 master1 message1 <--- START_TAG
20190101 receiver1 message1 <--- END_TAG
20190101 receiver2 message1 <--- END_TAG

When I parse my log only for one event is the elapsed_time calculated. All the other events get tagged with elapsed_end_without_start

My question: Is there any solution how I can calculate the elapsed_time for all of my messages?

My logstash config:

grok {
    match => { "SYSTEMID" => "master%{GREEDYDATA}" }
    add_tag => [ "taskStarted" ]
}

grok {
    match => { "SYSTEMID" => "receiver%{GREEDYDATA}" }
    add_tag => [ "taskFinished" ]
}

elapsed {
    start_tag => "taskStarted"
    end_tag => "taskFinished"
    unique_id_field => "signalID"
    timeout => 500
    new_event_on_match => false
}

Thank you for your help
Patrick

[Badger]

Instead of doing that with an elapsed filter I would do it using aggregate. Add the timestamp of the master1 message to the map[ ], then look it up for each receiver message and calculate the duration.

[kkulkarni (Kirtish)]

Hi Patrick,

I had similar use-case and was able to calculate the elapsed time. You need to add aggregate function.

grok {
match => { "SYSTEMID" => "master%{GREEDYDATA}" }
add_tag => [ "taskStarted" ]
}

grok {
match => { "SYSTEMID" => "receiver%{GREEDYDATA}" }
add_tag => [ "taskFinished" ]
}

elapsed {
start_tag => "taskStarted"
end_tag => "taskFinished"
unique_id_field => "signalID"
timeout => 500
new_event_on_match => false
}
if "in1" in [tags] and "elapsed" in [tags] {
aggregate {
task_id => "%{signalID}"
code => "map['report'] = [(event['elapsed_time']*1000).to_i]"
map_action => "create"
}
}