Layered continuous transforms?

Elastic Stack Elasticsearch
1

Is it possible to create a continuous transform with several processors that feeds into another continuous transform ?

2

As the output of a transform is another index, you can use the destination index of transform 1 as source for transform 2. To make this this continuous you need timestamps. 2 possible options:

  • use a min or max aggregation in transform 1 to write a timestamp into the destination of transform 1 which you can use as sync parameter for transform 2
  • use a ingest pipeline in transform 1 and set the ingest timestamp to a field which you can use for sync in transform 2

The next challenge is to configure transform 2 with the right delay, if you choose the ingest pipeline option. This should be simple: use the default (60s) or a lower one. The lowest possible delay: the ingest delay depending on your pipeline plus the refresh_interval of the index (default 1s).

If you took used an aggregation in transform 1 to create a timestamp you have to consider the delay, frequency etc. and think about the worst case timestamp that the output of transform 1 could be behind real time.

What I described is about what you can do today, however we are thinking about creating better pipelines. It's not only about connecting different transforms, the usecases we have in mind are also about connecting the output of a transform to data frame analytics. Transform in this usecase is the feature generator to eventually train a model with data frame analytics (and the model will than be used in inference).

Speaking about vision: Long term we plan continuous transforms without requiring a timestamp for synchronizing. In the case of a transform feeding to another transform, its thinkable that at the end of a checkpoint of transform 1 it triggers transform 2.

1 Like
3

Thank you Hendrik that makes sense.

This would be quite nice as it would reduce the legwork and code to reproduce in JS. A continuously trained ml model using layered continuous transforms. :blush:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.