Hello,
I am experimenting Transforms and I would to confirm that the limitations below are true:
- Doc removal detection in source and reflection upon next sync. If a doc is removed at source the value remains in the destination, however, if you add a doc in source index with the same aggregation keys the destination gets updated accordingly later.
- Dynamic destination index creation using field values like logstash does, I would assume one needs to create several transforms with filters to mimic this behavior.
- Option to associate a pipeline when creating the Transforms job (I believe I saw a field to define it, but I may be wrong)
I would like to take the opportunity to mention that it took me some attempts to understand what Frequency vs Delay really do because of the statement in quotes below. What is the objective of having a frequency ranging from 1s to 1h if the delay parameter is the one that truly does the changes in the destination index? I ask this because it makes more sense to have the Frequency and Delay being the same value, unless one wants to spread both more to save resources, also it is not clear which part costs more (checks or indexing)
Frequency
"In a continuous transform, the frequency configuration option sets the interval between checks for changes in the source indices. If changes are detected, then the source data is searched and the changes are applied to the destination index. Depending on your use case, you may wish to reduce the frequency at which changes are applied. By setting frequency to a higher value (maximum is one hour), the workload can be spread over time at the cost of less up-to-date data."
Thank you