So I am logging a bunch of logs of a big and mediumly complex workflow. This workflow is a graph and can take different paths based on the inputs or a particular workflow step's output. When the workflow is created a workflow id is generated, and is logged with the timestamp.
Now I am taksed with calculating the average time it takes for the workflow to reach a workflow step X. What is the best way to achieve this?
In my research so far, I have come across elapsed filter and this suggestion on a similar question asked previously here: Calculating duration between two documents in Elasticsearch
I feel elapsed is not suited for my use case as there are lot of workflow steps and we process great volumes of data every day and most workflows take hours to complete and in some cases days.
For the second option I couldn't find the script or a clear approach to do that. Any help would be greatly appreciated.