# Questions related to transforms limitations **URL:** \ **Category:** Elasticsearch\ **Tags:** transforms\ **Created:** [June 27, 2021, 6:28pm UTC](https://discuss.elastic.co/t/questions-related-to-transforms-limitations/277158 "2021-06-27T18:28:55Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![crpj](https://avatars.discourse-cdn.com/v4/letter/c/edb3f5/32.png) [@crpj](https://discuss.elastic.co/u/crpj)\ **Post date:** [June 27, 2021, 6:28pm UTC](https://discuss.elastic.co/t/questions-related-to-transforms-limitations/277158/1 "2021-06-27T18:28:55Z")
Hello, I am experimenting Transforms and I would to confirm that the limitations below are true: 1. Doc removal detection in source and reflection upon next sync. If a doc is removed at source the value remains in the destination, however, if you add a doc in source index with the same aggregation keys the destination gets updated accordingly later. 2. Dynamic destination index creation using field values like logstash does, I would assume one needs to create several transforms with filters to mimic this behavior. 3. Option to associate a pipeline when creating the Transforms job (I believe I saw a field to define it, but I may be wrong) I would like to take the opportunity to mention that it took me some attempts to understand what Frequency vs Delay really do because of the statement in quotes below. What is the objective of having a frequency ranging from 1s to 1h if the delay parameter is the one that truly does the changes in the destination index? I ask this because it makes more sense to have the Frequency and Delay being the same value, unless one wants to spread both more to save resources, also it is not clear which part costs more (checks or indexing) > Frequency > > "In a continuous transform, the frequency configuration option sets the interval between checks for changes in the source indices. If changes are detected, then the source data is searched and the changes are applied to the destination index. Depending on your use case, you may wish to reduce the frequency at which changes are applied. By setting frequency to a higher value (maximum is one hour), the workload can be spread over time at the cost of less up-to-date data." Thank you ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [June 28, 2021, 7:35am UTC](https://discuss.elastic.co/t/questions-related-to-transforms-limitations/277158/2 "2021-06-28T07:35:01Z")
> [@crpj](#): > > Doc removal detection in source and reflection upon next sync. ... Correct, deletes are invisible when detecting changes. > [@crpj](#): > > Dynamic destination index creation using field values like logstash does, Usually a transform has 1 destination index, however by writing into a ingest pipeline you can similar to logstash change the destination index by rewriting `_index`. If you do this, note that transform can't manage the index, but you have to manage it yourself. > [@crpj](#): > > Option to associate a pipeline when creating the Transforms job (I believe I saw a field to define it, but I may be wrong) Yes, see the [dest/pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-transform.html#put-transform-request-body). As for _frequency_ vs. _delay_: _Frequency_ control how often the transform runs, how often it gets updated. _Delay_ defines a time window, where data can arrive late due to various delays that happen during ingest and indexing, at minimum this is the refresh interval of Lucene indexes (default `1s`), other delays are time spend in ingest or logstash, time spent for sending data from the origin, etc. For example imagine an application sends telemetry data, for performance reasons the daemon flushes data every `10s`, afterwards the data gets queued(`2s`), processed (`2s`) and finally indexed (`1s`). _Delay_ is the sum: `(10 + 2 + 2 + 1)s`. If for some reason these `15s` are breached, transform might not see the update. To rule out potential problems, you can instead of using a timestamp value created externally use a _ingest timestamp_ that you set as last step in an ingest pipeline. In the example above, this would allow you to set _delay_ to e.g. `2s` to only account for the index refresh interval. Regarding resource usage: _Delay_ doesn't make any difference w.r.t. performance, because it is just an offset. However, `frequency` controls how often you check for updates, more recency costs performance. I agree _Frequency_ and _Delay_ are quite complex, we are working on simplifications for the future. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 26, 2021, 7:35am UTC](https://discuss.elastic.co/t/questions-related-to-transforms-limitations/277158/3 "2021-07-26T07:35:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.