Get duration between 2 dates in logs

[A_Chichi (Antoine)]

Hello,

I would like to get the duration of a job depending on my logs. I've checked this link : Documentation and this one Elastic forum which is exactly what I would like to do but I don't find the solution...

here is a sample of my log :

2021-01-14 17:01:27;26812;Job1;subjob1;begin;;
2021-01-14 17:01:42;26812;Job1;subjob1;end;success;

Here is the grok :

%{TIMESTAMP_ISO8601:date_log};%{INT:services_id};%{DATA:job};%{DATA:subjob};%{DATA:progress};%{DATA:status};

I would like to have this output :

{
  "services_id": "26812",
  "subjob": "subjob1",
  "date_log": "2021-01-14 17:01:42",
  "progress": "end",
  "job": "Job1",
  "status": "success",
  "duration": "15"
}

Here is what I've tried :

filter {
  grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:date_log};%{INT:services_id};%{DATA:job};%{DATA:subjob};%{DATA:progress};%{DATA:status};" }
  }

  date {
      match => [ "date_log", "yyyy-MM-dd HH:mm:ss" ]
  }

  if [progress] == "begin" {
    aggregate {
      task_id => "%{services_id}"
      code => "map['date_begin'] = event.get('date_log')"
      map_action => "create"
    }
  }

  if [progress] == "end" {
    aggregate {
      task_id => "%{services_id}"
      code => "
        require 'time'
        event.set('duration', Time.parse(event.get('date_log'))-Time.parse(map['date_begin']))"
      map_action => "update"
      end_of_task => true
      timeout => 120
    }
  }
}

This configuration gives me random 0 in some logs.

Sometimes I get "duration": "0" (which is not the case) and most of the time I get "duration": ""