How to use aggregate function in logstash to calculate difference between 'elapsed' time fields?

Elastic Stack Logstash
1

I am using the latest version of logstash(7.6.2). I am trying to calculate the total time spent in system excluding time spent between Out2 and In2 i.e. [(Out3-In1) - (Out2-In2)] for a given ID. Find below my sample data with columns Timestamp, Event, and ID

Sample Data

18/May/2015:02:05:10 +0000 In1 100
18/May/2015:02:05:15 +0000 Out1 100
18/May/2015:02:05:26 +0000 In2 100
18/May/2015:02:05:48 +0000 Out2 100
18/May/2015:15:05:08 +0000 In3 100
18/May/2015:16:05:03 +0000 Out3 100

I am currently using two elapsed functions and 2 aggregate functions as shown below in the config file

Config File

filter{
    grok{
        match => {"message" => "%{HTTPDATE:timestamp} %{WORD:Event} %{NUMBER:ID}"}
        add_tag => ["%{Event}"]
    }
    date{
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    elapsed {
        start_tag => "In1"
        end_tag => "Out3"
        unique_id_field => "ID"
    }
    elapsed {
        start_tag => "In2"
        end_tag => "Out2"
        unique_id_field => "ID"
    }
    if [Event] == "Out2" {
        aggregate {
            task_id => "%{ID}"
            code => "map['time1'] = (event.get('elapsed_time')*1000).to_i"
            map_action => "create"
        }
    }
    if [Event] == "Out3" {
        aggregate {
            task_id => "%{ID}"
            code => "map['time2'] = (event.get('elapsed_time')*1000).to_i - map['time1']; event.set('time1', map['time2'])"
            map_action => "update"
            end_of_task => true
        }
    }
}

I was able to generate elapsed time fields but was unable to calculate the difference between then using Agreggate filter. Find below my output

Is there anything I am missing?

Any help is appreciated! Thanks in advance

2

I have added piepline.workers:1 in the logstash.yml and it works now.

3

You may also need to disable java_execution until you get to 7.7.

4

@Badger Thanks for letting me know. But, where do I need to disable this? In the logstash.yml file?

5

You can do it in logstash.yml or on the command line.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.