How to parse a TimeSpan value which displays the difference between to timestamps

Elastic Stack Logstash
1

I'm trying to parse through a timespan value (hh:mm:ss:SSSSSS). Here is the message ...

`2020-08-05 13:28:53.128 +03:00 [DBG] {"Timestamp":"2020-08-05T13:28:53.1286882+03:00","Profile":"db","ProcessID":26688,"SessionId":"0","Message":"SELECT column1, column2 FROM hsi.table","TimeSpan":"00:00:00.0019088","LogLevel":"Debug",,"Direction":"Out","AppPoolName":"Server"}`

I would like to store the TimeSpan value as a sortable column. I looked at this link, but didn't follow how it worked and it didn't include the milliseconds.

My logstash.config uses the following filter settings.

filter {
    mutate{
        gsub => ["message", "^.{1,37}(.*)$","\1"]
    }
    json{
        source=> "message"      
    }  
    grok {
		match => { 
			 "TimeSpan" => { "%{INT:hours}:%{INT:minutes}:%{INT:seconds}.%{INT:miliseconds}" }
		}            
    }	
	date {
		match => [ "Timestamp", "ISO8601" ]
	}	
}

Thanks for your help.

2

That example predates the event API. You can no longer directly reference the event as a hash. These days it would be

    grok { match => { "TimeSpan" => "%{INT:hours}:%{INT:minutes}:%{INT:seconds}.%{INT:subsecond}" } }
    date { match => [ "Timestamp", "ISO8601" ] }
    ruby {
        code => '
            subsecond = event.get("subsecond")
            subsecond = subsecond.to_f / (10 ** subsecond.length)
            event.set("elapsed", 3600 * event.get("hours").to_f + 60 * event.get("minutes").to_f + event.get("seconds").to_f + subsecond)
        '
        remove_field => ["hours", "minutes", "seconds", "subsecond"]
    }

The 10 ** stuff for subsecond ensure that it can handle time spans like "01:02:03.2". Note that your JSON is not valid since it has ,, just after the LogLevel.

3

Thanks you @Badger. I'll give that a try.

The double ,, was a typo as I was cleaning up the JSON to post it. Good eyes. :wink:

4

Follow up... when I start Logstash I get the following error

Ruby exception occurred: undefined method 'length' for nil:NilClass

The error seems to be coming from this line

subsecond = subsecond.to_f / (10 ** subsecond.length)

Any thoughts on how I can resolve it?

5

You did update the grok to refer to subsecond instead of milisecond, right?

6

Yes, I updated it to the following ...

	grok { 
		match => { 
			"TimeSpan" => "%{INT:hours}:%{INT:minutes}:%{INT:seconds}.%{INT:subsecond}" 			
		} 
	}
7

That is telling you that the [subsecond] field does not exist. You could change the ruby code to

subsecond = event.get("subsecond")
if subsecond
    subsecond = subsecond.to_f / (10 ** subsecond.length)
    event.set("elapsed", 3600 * event.get("hours").to_f + 60 * event.get("minutes").to_f + event.get("seconds").to_f + subsecond)
end
8

Thanks @Badger. I'll give that a try. Appreciate the help.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.