Creating new scripted field to showing time difference between 2 log lines

[singyim817 (Sing Yim)]

Hi there,

I got a similar situation like this.

How to calculate time difference between two events in postprocessing?

Hi there, I have logs which contain a log line for starting and finishing a process. it looks like this: [image]. I need the time difference between the "tux state retrieved" and "tux state retrieved". Since the data is already indexed, I would like to know, if there is an option to calculate this in kibana or timelion as postprocessing. I am aware that there are possibilities of calculating it on index times in logstash. I will check this in future. Thanks, Andreas

I have assigned a unique ID for each pair of lines, then I need to calculate the transaction time in second between the datetime value in these lines, finally display on line chart to show average transaction time by date time.

Is there any possible way without modifying logstash config?

Thank you very much

[wylie (Wylie Conlon)]

Elasticsearch and Kibana generally expect you to work with a single document at a time, but there are some options you can use:

1. Without changing your logstash pipeline, you could try using the transformations feature of Elasticsearch, because you said that each pair of lines has a unique id. You could set up the pipeline this way:

• Group by unique ID
• Get the minimum timestamp and maximum timestamp and assign them to a single field
• I think you can even calculate the max - min timestamp using a painless script in the transformations

This would let you visualize the duration using Kibana.

2. Another option might be to use Vega to construct a query where you group by the unique IDs, and do the same calculation that I previously described. This is probably harder than the previous option.

[singyim817 (Sing Yim)]

Thanks for your answering @wylie .

For first solution, I can't find transformations feature from my Kibana.

May I make sure the above solutions are supported with my Kibana?

My Kibana version is 6.2.3

Thank you.

[wylie (Wylie Conlon)]

Transforms were introduced in 7.2, and have been significantly improved in later versions. 7.9 is the most recent version of the stack that has been released.