Scripted field: Get difference of between alternate occurrences of an event

[Ruthu_Ramya]

Hi,

I have a field in ES which holds timestamp like below:

Nov 30,14:49:56.785 INFO Plane 1A landed
Nov 30,14:49:57.099 INFO Plane 2B landed
Nov 30,14:49:57.368 INFO Plane 2C landed
Nov 30,14:49:57.620 INFO Plane 2D landed
Nov 30,14:49:57.900 INFO Plane 5X landed
Nov 30,14:49:58.267 INFO Plane 6G landed

I need the difference between the alternate occurrence.
Ex:

diff13 = Nov 30,14:49:57.368 - Nov 30,14:49:56.785 ( = 583ms)
diff24 = Nov 30,14:49:57.620 - Nov 30,14:49:57.099 ( = 521ms)

Any help in getting this done using scripted fileds in Kibana? Thanks.

Regards,
Ruthu

[asp]

Hi,

do you have multiple timestamp fields in the same event, or do you need to calculate between different events?

[Ruthu_Ramya]

Hi @asp,

multiple timestamp fields in the same event.

[asp]

you have two options here as I see it:

1. do it in logstash:

• you can use a ruby filter in logstash. Here is one example:

ruby
    	{
    		init => 
    		"
    			require 'time'
    		"
    		code => 
    		"
    			currentTimeInt=DateTime.now.strftime('%Q').to_i
    			event.set('[logstash][processing][filterEnd]', currentTimeInt)
    			event.set('[logstash][processing][filterTime]',(currentTimeInt - event.get('[logstash][processing][filterStart]')))
    		"
    	}

Benefit on using logstash:

• you can filter on the field
• you can aggregate on the field

Disadvantage: you are kinda static because you have to decide / code it before your data runs in.

2. use scripted fields:
   I haven't done it with timestamps yet, but you should be able to calculate by scripted fields.
   example for numeric values:

doc['system.cpu.system.pct'].value + doc['system.cpu.user.pct'].value

The result is computed at each query runtime, so it is more cpu expensive.
You cannot filter / aggregate on a scripted field.

But the advantage is, you can have the new field available, even for old data.

[Ruthu_Ramya]

@asp,

Thanks for your response, After reading your reply, I would prefer to do it in logstash itself.
I think my original question was not clear enough, I have updated the log lines for clarity. Assume, that is how the log lines appear. I need the difference of the timestamps mentioned in the loglines(alternate) .

I do not know Ruby, but what I understood is that the code above will give the actual time difference between the log events. Please correct me if i am wrong.

Any help here?

[asp]

For debugging my filters I created two filters in logstash.
The first filter which is processed creates a field logstash.processing.filterStart with the current time.
The last filter which is processed contains the part I posted above. It creates a new field logstash.processing.filterEnd and it calculates the time difference between these two fields, which are both stored in the same event.

For your example as I understand you have multiple events = log lines and you want to diff between them.
How this works, I just want to know by myself, but unfortunately I don't.

In my case both timestamps are fields in the same event which was created out of a single log line.