Is it possible to create a continuous transform with several processors that feeds into another continuous transform ?
As the output of a transform is another index, you can use the destination index of transform 1 as source for transform 2. To make this this continuous you need timestamps. 2 possible options:
- use a
maxaggregation in transform 1 to write a timestamp into the destination of transform 1 which you can use as
syncparameter for transform 2
- use a ingest pipeline in transform 1 and set the ingest timestamp to a field which you can use for
syncin transform 2
The next challenge is to configure transform 2 with the right
delay, if you choose the ingest pipeline option. This should be simple: use the default (
60s) or a lower one. The lowest possible
delay: the ingest delay depending on your pipeline plus the
refresh_interval of the index (default
If you took used an aggregation in transform 1 to create a timestamp you have to consider the
frequency etc. and think about the worst case timestamp that the output of transform 1 could be behind real time.
What I described is about what you can do today, however we are thinking about creating better pipelines. It's not only about connecting different transforms, the usecases we have in mind are also about connecting the output of a transform to data frame analytics. Transform in this usecase is the feature generator to eventually train a model with data frame analytics (and the model will than be used in inference).
Speaking about vision: Long term we plan continuous transforms without requiring a timestamp for synchronizing. In the case of a transform feeding to another transform, its thinkable that at the end of a checkpoint of transform 1 it triggers transform 2.
Thank you Hendrik that makes sense.
This would be quite nice as it would reduce the legwork and code to reproduce in JS. A continuously trained ml model using layered continuous transforms.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.