Scripted field: Get difference of between alternate occurrences of an event

Hi,

I have a field in ES which holds timestamp like below:

Nov 30,14:49:56.785 INFO Plane 1A landed
Nov 30,14:49:57.099 INFO Plane 2B landed
Nov 30,14:49:57.368 INFO Plane 2C landed
Nov 30,14:49:57.620 INFO Plane 2D landed
Nov 30,14:49:57.900 INFO Plane 5X landed
Nov 30,14:49:58.267 INFO Plane 6G landed

I need the difference between the alternate occurrence.
Ex:

diff13 = Nov 30,14:49:57.368 - Nov 30,14:49:56.785 ( = 583ms)
diff24 = Nov 30,14:49:57.620 - Nov 30,14:49:57.099 ( = 521ms)

Any help in getting this done using scripted fileds in Kibana? Thanks.

Regards,
Ruthu

Hi,

do you have multiple timestamp fields in the same event, or do you need to calculate between different events?

Hi @asp,

multiple timestamp fields in the same event.

you have two options here as I see it:

  1. do it in logstash:
  • you can use a ruby filter in logstash. Here is one example:
ruby
    	{
    		init => 
    		"
    			require 'time'
    		"
    		code => 
    		"
    			currentTimeInt=DateTime.now.strftime('%Q').to_i
    			event.set('[logstash][processing][filterEnd]', currentTimeInt)
    			event.set('[logstash][processing][filterTime]',(currentTimeInt - event.get('[logstash][processing][filterStart]')))
    		"
    	}

Benefit on using logstash:

  • you can filter on the field
  • you can aggregate on the field

Disadvantage: you are kinda static because you have to decide / code it before your data runs in.

  1. use scripted fields:
    I haven't done it with timestamps yet, but you should be able to calculate by scripted fields.
    example for numeric values:

doc['system.cpu.system.pct'].value + doc['system.cpu.user.pct'].value

The result is computed at each query runtime, so it is more cpu expensive.
You cannot filter / aggregate on a scripted field.

But the advantage is, you can have the new field available, even for old data.

@asp,

Thanks for your response, After reading your reply, I would prefer to do it in logstash itself.
I think my original question was not clear enough, I have updated the log lines for clarity. Assume, that is how the log lines appear. I need the difference of the timestamps mentioned in the loglines(alternate) .

I do not know Ruby, but what I understood is that the code above will give the actual time difference between the log events. Please correct me if i am wrong.

Any help here?

For debugging my filters I created two filters in logstash.
The first filter which is processed creates a field logstash.processing.filterStart with the current time.
The last filter which is processed contains the part I posted above. It creates a new field logstash.processing.filterEnd and it calculates the time difference between these two fields, which are both stored in the same event.

For your example as I understand you have multiple events = log lines and you want to diff between them.
How this works, I just want to know by myself, but unfortunately I don't.

In my case both timestamps are fields in the same event which was created out of a single log line.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.