Hi there,
So I have reams of different types of time series data. Currently i've chosen to put each type of data into their own index because with the exception of 4 fields, all of the data is very different. Also the data is sampled at different rates and are not guaranteed to have common timestamps across the same sub-second window so fusing them all into one large document is also not a trivial task.
One of our common use cases that i'm trying to see if I can solve entirely in ES is to return an aggregation result of one index based on the time windows returned from a query of another index. Pictorially:
This is what I want to accomplish. for small enough signal transitions on the "condition" data, I can just use a date histogram and some combination of a top hits sub aggregation, but this quickly breaks down when i have 10,000's or 100,000's of occurrences of "the condition". Further this is just one "case", i have 100's of sets of similar situations that i'd like to get the overall min/max from.
The comparisons are basically amongst what I would consider to be sibling level documents or indices, so there doesn't seem to be any obvious parent->child relationship that would be flexible enough over the long run, at least with how the data is currently structured.
It feels like there should be an elegant solution instead of brute force building the date ranges outside of ES with the results of one query and feeding 100's of time ranges into another query. I've been pouring the the documentation all day today ( still fairly green at queries and agg's in ES ) and it feels like some combination of ES scripting and some of the pipelined aggregations are going to be what i want, but no definitive solutions are jumping out at me. I could really use some pointers in the right direction from the community.
Thanks