Duration calculation using date fields

Elastic Stack Logstash
1

We have log file in which we have to capture the first line matching "TIMESTAMP_ISO8601" against build_StartTime filed and last line matching "TIMESTAMP_ISO8601" against build_EndTime filed. After this we have to calculate the difference and should store the difference against build_Duration filed. But we are getting below error. Hope Build_StartTime & build_EndTime is not stored as date field rather it's stored as string field. please help us
Log File:
[2019-01-31 21:28:22Z INFO Program] Version: 2.122.1
some logs in the middle
[2019-01-31 21:28:45Z INFO Worker] Job completed.
Below is our current Logstash config File:
input {
beats {
client_inactivity_timeout => 1200
port => 5002
}
}

filter
{
if [message] =~ "Version: 2.122.1"
{
grok {
add_tag => [ "start" ]
match => { "message" => "%{TIMESTAMP_ISO8601:build_StartTime}" }
}
date {
add_tag => [ "start" ]
match => ["build_StartTime", "ISO8601"]
target => "build_StartTime"
}
}
grok {
add_tag => [ "start" ]
break_on_match => false
match => {
"message" => [
'"..definitionName": "(?<build_DefinitionName>.?)"',
'"..requestedFor": "(?<build_RequesterName>.?)"'
]a
}
}

if [message] =~ "Job result after all post-job steps finish:"
{
	grok {
		add_tag => [ "start" ]
		match => { "message" => "%{TIMESTAMP_ISO8601:build_EndTime}" }
	}
	date {
		match => ["build_EndTime", "ISO8601"]
		target => "build_EndTime"
	}
	grok {
		add_tag => ["end"]
		match => { "message" => "Job result after all post-job steps finish:(?<build_Status>.([A-Za-z]*))" }
	}
	if ![build_Status] or [build_Status] == " "
	{
		mutate {
		add_tag => [ "start" ]
		update => { "build_Status" => "Succeeded" }
		}
	}
	ruby {
	      add_tag => [ "start" ]
	     init => "require 'time'" 
	     code => "event.set('build_Duration', [event.get('build_EndTime') - event.get('build_StartTime')])"
	}

}
if "start" in [tags] {
	aggregate {
	task_id => "%{source}"
		code => "
		map['build_DefinitionName'] = event.get('build_DefinitionName') unless event.get('build_DefinitionName').nil?
		map['build_RequesterName'] = event.get('build_RequesterName') unless event.get('build_RequesterName').nil?
		map['build_StartTime'] = event.get('build_StartTime') unless event.get('build_StartTime').nil?
		map['build_EndTime'] = event.get('build_EndTime') unless event.get('build_EndTime').nil?
		map['build_Status'] = event.get('build_Status') unless event.get('build_Status').nil?
		map['build_Duration'] = event.get('build_Duration') unless event.get('build_Duration').nil?
		"
	}
}

if "end" in [tags] {
	aggregate {
	task_id => "%{source}"
		code => "
		event.set('build_DefinitionName', map['build_DefinitionName'])
		event.set('build_RequesterName', map['build_RequesterName'])
		event.set('build_StartTime', map['build_StartTime'])
		event.set('build_EndTime', map['build_EndTime'])
		event.set('build_Status', map['build_Status'])
		event.set('build_Duration', map['build_Duration'])
		"
	end_of_task => true
	}
}
if "end" not in [tags] or [build_DefinitionName] == 'nil' or ![build_DefinitionName] {
drop { }
}
mutate {
remove_field => [ "message" ]
}
mutate {
remove_tag => [ "start" ]
}

}

output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "tfslog-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

Output:
[2019-02-21T12:13:38,484][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `-' for "2019-01-31 21:28:22Z":String
Did you mean? -@

2

Please do not post the same question 4 times.

3

Sorry my entire team using the same credentials for Community Login. I am not aware that there is an other topic opened with the question from my team. Please ignore others and help me here

4

Which Logstash version do you use ?

Can you say if start time and end times are in the same log line or in two different log lines ?

Can you provide some example input log line(s) ?

5

Given your error, you shoud replace :
code => "event.set('build_Duration', [event.get('build_EndTime') - event.get('build_StartTime')])"
by :
code => "event.set('build_Duration', event.get('build_EndTime').time - event.get('build_StartTime').time)"

6

Thanks for the suggestions Fabien Baligand. Actually I can fix the issue by changing my code line below
if [message] =~ "Version: 2.122.1"
{
grok {
match => { "message" => "%{DATESTAMP:build_StartTime}" }
}
date {
match => [ "build_StartTime", "ISO8601" ]
target => "build_StartTime"
}
}

by:

    if [message] =~ "Version: 2.122.1"
    {
            grok {
            match => { "message" => "%{DATESTAMP:build_StartTime}" }
            }
            date {
            match => [ "build_StartTime", "yy-MM-dd HH:mm:ss" ]
            target => "build_StartTime"
            }
    }

The problem in my build_StartTime is in this format "19-01-31 21:28:22" for which I was using ISO8601 which is totally wrong for which I must use yy-MM-dd HH:mm:ss. Now I can get the field as date. Thanks again

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.