Aggregation to find largest range (diff of min max)

[smlbiobot (SML)]

Is there a way to quickly find the largest range of values (i.e. the difference of min and max within a timeframe)?

What I want to do: in a time series with an example doc structured as something like below, find the product with the greatest price fluctuation over specific period of time:

{
    "product": "apple",
    "price": 1.04,
    "timestamp": "2021-02-28T03:01:07+00:00"
}

I know that I can find the min and max value through aggregations and then use the results to find the largest range, but doing so would require me to first perform aggregation to every single product and do calculations afterwards. This takes time. My actual use case involves millions of categories (they are actually not prices of food), and I’d like to do this in real-time. So I’m wondering if there is other types of aggregation that will do this that I’m not aware.

Thanks!

[nik9000 (Nik Everett)]

I don't think there is a great way to get the thing you want without, say resorting specific routing values and scripted metric or writing a plugin to make a new agg. The tricky bit is that without routing all of the prices for a particular food item to the same shard you can't get an accurate range on the data node. And if you can't do that you have to send too all of the results back to the data node. I'd you route the data together you can take advantage of that. But none of the built in aggs do that.

[smlbiobot (SML)]

Thank you. This is as much as I expected.

Do you know if there is a way to do aggregation in a progressive way? For example: if I want to get results in a search, I can use Scan. But is there an equivalent to that for aggregation? (Say I get the max for all items and then get the min for all items and then do this calculation for range myself)

The reason I ask is because I don’t want the aggregation to stall the cluster (if that makes any sense). I have seen that doing very large aggregations like this could have this effect, and as I do have other things running I need to give it room to perform other tasks besides doing this one calculation.