For a transform you need 2 parts: the group_by and the aggregation part. It seems to me that you want to group by job, buildapp1, buildapp2 etc.
As you want to reduce stages as well it seems logical to group by message, however this collides with you goal to get the value from the previous message. If you group by message the Stage:Build event is not able to see the output 4 event. Therefore I suggest to skip grouping on message. I think it should still work for your visualization if the output document looks like this:
{
"job": "buildapp1",
"timestamp": 20210202...,
"duration" : {
"checkout": 153,
"build": 205
}
}
I think that having a consistent timestamp (start of the build) is better than having 2 timestamps, if not, you could also output 2 timestamps. (my timestamp is just min(timestamp))
To get the desired fields: checkout and build there is no other way than to script it. The scripted_metric aggregations gives you that flexibility. In a nutshell you first collect all messages and than process them. I would use a SortedMap and collect the events in the map script, keyed by timestamp. Note: the map script is executed for every shard. The main logic goes into the reduce script. You need to merge all the maps. Afterwards you can iterate the collected data in sorted order. A variable remembers the last value, so you can access it in the next iteration. The rest is text matching on the message, e.g. if the message contains Build you calculate the duration based on the current and last value. Finally you want to output the fields: a timestamp (from the 1st value?) and the checkout duration, build duration etc. The way to output multiple values is using (again) a Map, e.g. HashMap with fieldnames as key [*].
I hope this gives you an idea, the implementation requires some work and probably some trial and error if you haven't worked with painless yet. For this special type of work, it might be more efficient to use the dev console and the _preview API from transform until you found the script.
The documentation does not contain an example that fits, but maybe it still helps to take a look.
