Calculate time difference between two different record timestamps

So I am logging a bunch of logs of a big and mediumly complex workflow. This workflow is a graph and can take different paths based on the inputs or a particular workflow step's output. When the workflow is created a workflow id is generated, and is logged with the timestamp.

Now I am taksed with calculating the average time it takes for the workflow to reach a workflow step X. What is the best way to achieve this?

In my research so far, I have come across elapsed filter and this suggestion on a similar question asked previously here: Calculating duration between two documents in Elasticsearch

I feel elapsed is not suited for my use case as there are lot of workflow steps and we process great volumes of data every day and most workflows take hours to complete and in some cases days.

For the second option I couldn't find the script or a clear approach to do that. Any help would be greatly appreciated.

I updated the video comments with a version I just updated to work with elasticsearch 6.3 -

1 Like

Thanks, Mark! I will give it a shot and revert if I run into issues.

And btw, great talks on Youtube!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.