Combining two events in one to calculate time difference

[mangeshmj1992 (Mangesh Jadhav)]

Hello team,
I have case where information is being displayed in different lines. These lines are not even consecutive lines. Unique field is log_processed.companyId. I have to display time difference between entry and exit log.

I have tried with aggregate filter but it didn't work

Sample log:

{"timestamp":"2021-11-17T05:52:59:613","level":"info","message":"ENTRY: INTERNAL_HTTP_REQUEST","userId":"rahul","companyId":"abc"}

{"timestamp":"2021-11-17T05:52:59:768","level":"info","message":"EXIT: INTERNAL_HTTP_REQUEST","userId":"rahul","companyId":"abc"}

Aggregate filter:

if [log_processed.message] == "ENTRY: INTERNAL_HTTP_REQUEST" {
        aggregate {
            task_id => "%{log_processed.companyId}"
            code => "map['started'] = event['@timestamp']"
            map_action => "create"
        }
    }

    if [log_processed.message] == "EXIT: INTERNAL_HTTP_REQUEST" {
        aggregate {
            task_id => "%{log_processed.companyId}"
            code => "event['duration'] = event['@timestamp'] - map['started']"
            map_action => "update"
            push_map_as_event_on_timeout => true
            timeout_task_id_field => "log_processed.companyId"
            timeout => 60 # 1 minutes timeout
			timeout_tags => ['_aggregatetimeout']
        }
    }

[mangeshmj1992 (Mangesh Jadhav)]

Hello @Badger ,
Can you please help me here

[stevedearl (Steve Earl)]

Hi @mangeshmj1992 ,

You should take a look at the Elapsed Filter as from your description that does what you're looking for.

We have 'start' and 'end' events (separate documents being fed by Logstash into Elasticsearch). They both contain a unique ID so it's possible to correlate which 'end' event relates to which 'start' event.
The elapsed plugin can monitor for these events, correlate them together and provide the 'elapsed time' by subtracting the start event timestamp from the end event timestamp.

Note that it's critical to have a unique correlation ID that both events contain.

Hope that helps,
Steve