Logstash - Difference between date into new field

[stemons]

Hello team,
I'm trying to add a new field to metricbeat data collection through logstash. The idea is to create a field from the difference between two dates (@timestamp and system.process.cpu.start_date).

I've made some test, but the field created in elastic is empty.

Here is the logstash configuration

input {
  beats {
    port => 5044
  }
}
filter {
  date {
         match => [ "@timestamp", "ISO8601" ]
         target => "start_date"
  }
  date {
         match => [ "system.process.cpu.start_time", "ISO8601" ]
         target => "end_date"
  }
  ruby {
        init => "require 'time'"
        code => "
                 duration = (event.get('start_date') - event.get('end_date')) rescue nil;
                 event.set('system.process.cpu.duration', duration);
                "
  }
}

[Iker (Iker Saint)]

I haven't tried your code but the start date, must be subtracted from the finish date, try with this, haven't tested but should be fine:

duration = event.get('end_date').to_i - event.get('start_date').to_i 

There is also a math filter for logstash if you like to try it.

GitHub - logstash-plugins/logstash-filter-math: This plugin provides the...

[stemons]

@stemons

ruby {
        init => "require 'time'"
        code => "
                 duration = (event.get('start_date') - event.get('end_date')) rescue nil;
                 event.set('system.process.cpu.duration', duration);
                "
  }

Using this now I'm getting always 0. What should be the result unit ? days?

system.process.cpu.duration 0
@timestamp Sep 9, 2021 @ 23:21:08.700
system.process.cpu.start_time Sep 9, 2021 @ 23:19:44.000

date {
         match => [ "@timestamp", "ISO8601" ]
         target => "end_date"
  }
  date {
         match => [ "system.process.cpu.start_time", "ISO8601" ]
         target => "start_date"
  }
ruby {
        init => "require 'time'"
        code => "
                 duration = (event.get('end_date').to_i - event.get('start_date').to_i) rescue nil;
                 event.set('system.process.cpu.duration', duration);
                "
  }

[Badger]

@stemons

system.process.cpu.start_time

logstash does not use the same syntax to refer to objects nested inside objects that filebeat documentation and elasticsearch use. In logstash that field would be referred to as

[system][process][cpu][start_time]

This allows logstash to unambiguously refer to fields that have . in their name.

[stemons]

Thanks. Now I'm getting epoch time format, but only for the field @timestamp.

For the other field I'm getting a wrong value (2021). Seems it's extracting the year from the date.

date_end_i
1631224028

date_start_i
2021

ruby {
        init => "require 'time'"
        code => "
                 duration = (event.get('@timestamp').to_i - event.get('[process][cpu][start_time]').to_i) rescue nil;
                 date_end_i = event.get('@timestamp').to_i;
                 date_start_i = event.get('[process][cpu][start_time]').to_i;
                 event.set('date_end_i',date_end_i);
                 event.set('date_start_i',date_start_i);
                 event.set('system.process.cpu.duration', duration);
                "
  }

[Badger]

@stemons

date_start_i = event.get('[process][cpu][start_time]').to_i;

That is going to take a string like "2021-03-29T04:24:52.000Z" and .to_i will just pull the leading 2021 from it. You need to parse that using a date filter to convert it to a LogStash::Timestamp (that's what a date filter always produces) so that .to_i returns a number of seconds.

[stemons]

@stemons

date {
         match => [ "system.process.cpu.start_time", "ISO8601" ]
         target => "end_date"
  }

It worked! Thanks a lot