# Layered continuous transforms?

**URL:** https://discuss.elastic.co/t/layered-continuous-transforms/229507
**Category:** Elasticsearch
**Tags:** elastic-stack-machine-learning
**Created:** [April 23, 2020, 3:35pm UTC](https://discuss.elastic.co/t/layered-continuous-transforms/229507 "2020-04-23T15:35:49Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Cain310](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cain310/32/53115_2.png) [@Cain310](https://discuss.elastic.co/u/Cain310)
#### Post date: [April 23, 2020, 3:35pm UTC](https://discuss.elastic.co/t/layered-continuous-transforms/229507/1 "2020-04-23T15:35:50Z")

</div>

Is it possible to create a continuous transform with several processors that feeds into another continuous transform ?

---

<div class="post-metadata">

### Author: ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [April 23, 2020, 6:13pm UTC](https://discuss.elastic.co/t/layered-continuous-transforms/229507/2 "2020-04-23T18:13:20Z")

</div>

As the output of a transform is another index, you can use the destination index of transform 1 as source for transform 2. To make this this continuous you need timestamps. 2 possible options:

- use a `min` or `max` aggregation in transform 1 to write a timestamp into the destination of transform 1 which you can use as `sync` parameter for transform 2
- use a ingest pipeline in transform 1 and set the ingest timestamp to a field which you can use for `sync` in transform 2

The next challenge is to configure transform 2 with the right `delay`, if you choose the ingest pipeline option. This should be simple: use the default (`60s`) or a lower one. The lowest possible `delay`: the ingest delay depending on your pipeline plus the `refresh_interval` of the index (default `1s`).

If you took used an aggregation in transform 1 to create a timestamp you have to consider the `delay`, `frequency` etc. and think about the worst case timestamp that the output of transform 1 could be behind real time.

What I described is about what you can do today, however we are thinking about creating better pipelines. It's not only about connecting different transforms, the usecases we have in mind are also about connecting the output of a transform to data frame analytics. Transform in this usecase is the feature generator to eventually train a model with data frame analytics (and the model will than be used in inference).

Speaking about vision: Long term we plan continuous transforms without requiring a timestamp for synchronizing. In the case of a transform feeding to another transform, its thinkable that at the end of a checkpoint of transform 1 it triggers transform 2.

---

<div class="post-metadata">

### Author: ![Cain310](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cain310/32/53115_2.png) [@Cain310](https://discuss.elastic.co/u/Cain310)
#### Post date: [April 28, 2020, 2:55pm UTC](https://discuss.elastic.co/t/layered-continuous-transforms/229507/3 "2020-04-28T14:55:21Z")

</div>

Thank you Hendrik that makes sense.

> [@Hendrik\_Muhs](#):
>
> Transform in this usecase is the feature generator to eventually train a model with data frame analytics (and the model will than be used in inference).

This would be quite nice as it would reduce the legwork and code to reproduce in JS. A continuously trained ml model using layered continuous transforms. 😊

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 26, 2020, 3:01pm UTC](https://discuss.elastic.co/t/layered-continuous-transforms/229507/4 "2020-05-26T15:01:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
