How to extract the time elapsed from a value of a field to another value of the same field

Elastic Stack Logstash
9

The file input, dissect and mutate look fine. I would not use a split filter at that point because that creates separate events, and you lose any association between events on the same line. Instead use a mutate and split to create an array that you can work on.

mutate {
    split => { "actions" => "|" }
    remove_field => ["message"]
}

grok { match => { "actions" => [ "%{WORD:action};%{DATA:ts};(?<user>[^;]*)$" ] } }

That grok takes the array and matches each entry. It returns three arrays - action, ts, and user. The last entry on each line does not have a user after the ;, so that array will have one less entries than the other arrays.

Although grok works on array, the date filter does not. You cannot pass it an array of strings and get back an array of dates in target, so we dive down into ruby that this point.

ruby {
    code => "

I use " to surround the code and ' within it. That's a personal preference, some folks prefer it the other way around. First we iterate over the ts array that grok returned and parse the timestamps.

        ts = event.get('ts')
        timestamps = []
        if ts then
            ts.each { |x|
                timestamps << Time.strptime(x, '%Y-%m-%d %H:%M:%S')
            }
        end
        # event.set('timestamps', timestamps)

Then for each timestamp except the first we calculate the delta. We get back a number in seconds.

        deltas = []
        timestamps.each_index { |x|
            if x == 0 then
                deltas << 0
            else
                deltas << timestamps[x].to_i - timestamps[x-1].to_i
            end
        }
        # event.set('deltas', deltas)

Now we have arrays for action, user, timestamp, and delta. It is probably useful to combine them. You could trivially add tienda or solicitud or other fields from the main event to each object.

        tienda = event.get('tienda')
        action = event.get('action')
        user = event.get('user')
        a = []
        action.each_index { |k|
            n = {
                'action'    => action[k],
                'user'      => user[k],
                'delta'     => deltas[k],
                'timestamp' => timestamps[k],
                'tienda'    => tienda
            }
            a << n
        }
        event.set('arrayOfObjects', a)

Here ends the ruby filter.

        # event.remove('ts')
        # event.remove('user')
        # event.remove('action')
        # event.remove('actions')
    "
}

At this point you may want to use a split on arrayOfObjects, since each object now has all the data in it.

"arrayOfObjects" => [
    [0] {
           "action" => "Creacion",
             "user" => "baseuser",
            "delta" => 0,
        "timestamp" => 2018-09-16T19:24:15.000Z,
           "tienda" => "0498"
    },
    [1] {
           "action" => "Tienda",
             "user" => "MLPARRAF",
            "delta" => 0,
        "timestamp" => 2018-09-16T19:24:15.000Z,
           "tienda" => "0498"
    },
    [2] {
           "action" => "Capturista",
             "user" => nil,
            "delta" => 1132,
        "timestamp" => 2018-09-16T19:43:07.000Z,
           "tienda" => "0498"
    }
1 Like