# Duration calculation using date fields

**URL:** https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407
**Category:** Logstash
**Created:** [February 21, 2019, 12:16pm UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407 "2019-02-21T12:16:13Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)
#### Post date: [February 21, 2019, 12:16pm UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/1 "2019-02-21T12:16:14Z")

</div>

We have log file in which we have to capture the first line matching "TIMESTAMP\_ISO8601" against build\_StartTime filed and last line matching "TIMESTAMP\_ISO8601" against build\_EndTime filed. After this we have to calculate the difference and should store the difference against build\_Duration filed. But we are getting below error. Hope Build\_StartTime & build\_EndTime is not stored as date field rather it's stored as string field. please help us  
**Log File:**  
[2019-01-31 21:28:22Z INFO Program] Version: 2.122.1  
some logs in the middle  
[2019-01-31 21:28:45Z INFO Worker] Job completed.  
**Below is our current Logstash config File:**  
input {  
beats {  
client\_inactivity\_timeout =\> 1200  
port =\> 5002  
}  
}

filter  
{  
if [message] =~ "Version: 2.122.1"  
{  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:build\_StartTime}" }  
}  
date {  
add\_tag =\> ["start"]  
match =\> ["build\_StartTime", "ISO8601"]  
target =\> "build\_StartTime"  
}  
}  
grok {  
add\_tag =\> ["start"]  
break\_on\_match =\> false  
match =\> {  
"message" =\> [  
'"..definitionName": "(?\<build\_DefinitionName\>.?)"',  
'"..requestedFor": "(?\<build\_RequesterName\>.?)"'  
]a  
}  
}

```
if [message] =~ "Job result after all post-job steps finish:"
{
	grok {
		add_tag => ["start"]
		match => { "message" => "%{TIMESTAMP_ISO8601:build_EndTime}" }
	}
	date {
		match => ["build_EndTime", "ISO8601"]
		target => "build_EndTime"
	}
	grok {
		add_tag => ["end"]
		match => { "message" => "Job result after all post-job steps finish:(?<build_Status>.([A-Za-z]*))" }
	}
	if ![build_Status] or [build_Status] == " "
	{
		mutate {
		add_tag => ["start"]
		update => { "build_Status" => "Succeeded" }
		}
	}
	ruby {
	      add_tag => ["start"]
	     init => "require 'time'" 
	     code => "event.set('build_Duration', [event.get('build_EndTime') - event.get('build_StartTime')])"
	}

}
if "start" in [tags] {
	aggregate {
	task_id => "%{source}"
		code => "
		map['build_DefinitionName'] = event.get('build_DefinitionName') unless event.get('build_DefinitionName').nil?
		map['build_RequesterName'] = event.get('build_RequesterName') unless event.get('build_RequesterName').nil?
		map['build_StartTime'] = event.get('build_StartTime') unless event.get('build_StartTime').nil?
		map['build_EndTime'] = event.get('build_EndTime') unless event.get('build_EndTime').nil?
		map['build_Status'] = event.get('build_Status') unless event.get('build_Status').nil?
		map['build_Duration'] = event.get('build_Duration') unless event.get('build_Duration').nil?
		"
	}
}

if "end" in [tags] {
	aggregate {
	task_id => "%{source}"
		code => "
		event.set('build_DefinitionName', map['build_DefinitionName'])
		event.set('build_RequesterName', map['build_RequesterName'])
		event.set('build_StartTime', map['build_StartTime'])
		event.set('build_EndTime', map['build_EndTime'])
		event.set('build_Status', map['build_Status'])
		event.set('build_Duration', map['build_Duration'])
		"
	end_of_task => true
	}
}
if "end" not in [tags] or [build_DefinitionName] == 'nil' or ![build_DefinitionName] {
drop { }
}
mutate {
remove_field => ["message"]
}
mutate {
remove_tag => ["start"]
}

```

}

output {  
elasticsearch {  
hosts =\> ["elasticsearch:9200"]  
index =\> "tfslog-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}

**Output:**  
[2019-02-21T12:13:38,484][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `-' for "2019-01-31 21:28:22Z":String  
Did you mean? -@

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 21, 2019, 2:44pm UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/2 "2019-02-21T14:44:37Z")

</div>

Please do not post the same question 4 times.

---

<div class="post-metadata">

### Author: ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)
#### Post date: [February 22, 2019, 5:13am UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/3 "2019-02-22T05:13:07Z")

</div>

Sorry my entire team using the same credentials for Community Login. I am not aware that there is an other topic opened with the question from my team. Please ignore others and help me here

---

<div class="post-metadata">

### Author: ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)
#### Post date: [February 22, 2019, 9:20am UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/4 "2019-02-22T09:20:34Z")

</div>

Which Logstash version do you use ?

Can you say if start time and end times are in the same log line or in two different log lines ?

Can you provide some example input log line(s) ?

---

<div class="post-metadata">

### Author: ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)
#### Post date: [February 22, 2019, 9:45am UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/5 "2019-02-22T09:45:17Z")

</div>

Given your error, you shoud replace :  
`code => "event.set('build_Duration', [event.get('build_EndTime') - event.get('build_StartTime')])"`  
by :  
`code => "event.set('build_Duration', event.get('build_EndTime').time - event.get('build_StartTime').time)"`

---

<div class="post-metadata">

### Author: ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)
#### Post date: [February 22, 2019, 10:06am UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/6 "2019-02-22T10:06:39Z")

</div>

Thanks for the suggestions Fabien Baligand. Actually I can fix the issue by changing my code line below  
if [message] =~ "Version: 2.122.1"  
{  
grok {  
match =\> { "message" =\> "%{DATESTAMP:build\_StartTime}" }  
}  
date {  
match =\> ["build\_StartTime", "ISO8601"]  
target =\> "build\_StartTime"  
}  
}

by:

```
    if [message] =~ "Version: 2.122.1"
    {
            grok {
            match => { "message" => "%{DATESTAMP:build_StartTime}" }
            }
            date {
            match => ["build_StartTime", "yy-MM-dd HH:mm:ss"]
            target => "build_StartTime"
            }
    }

```

The problem in my build\_StartTime is in this format "19-01-31 21:28:22" for which I was using ISO8601 which is totally wrong for which I must use yy-MM-dd HH:mm:ss. Now I can get the field as date. Thanks again

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 22, 2019, 10:06am UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407/7 "2019-03-22T10:06:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
