# Using transforms and ingest pipelines on data that changes over time
**URL:** https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101
**Category:** Elasticsearch
**Created:** [November 14, 2023, 9:48am UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101 "2023-11-14T09:48:46Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@pulsy](https://discuss.elastic.co/u/pulsy)
#### Post date: [November 14, 2023, 9:48am UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/1 "2023-11-14T09:48:46Z")
I'm thinking about using elastic transforms together with ingest pipelines to basically create views of mongodb collections that are spread over multiple database servers, so we can efficiently sort and filter by references over those databases.
My main question is: If we have a transform pipeline, with an ingest pipeline using an enrich processor on the destination index, is it possible to do a update-by-query call on the desination index once a reference changed so the enrich processor runs again on matching documents?
---
### Author:  [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)
#### Post date: [November 14, 2023, 11:18am UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/2 "2023-11-14T11:18:37Z")
Hi @pulsy,
You can use an ingest pipeline with an update\_by\_query, [as per this example](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-api-ingest-pipeline). Is that what you're thinking of?
---
### Author:  [@pulsy](https://discuss.elastic.co/u/pulsy)
#### Post date: [November 14, 2023, 11:57am UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/3 "2023-11-14T11:57:30Z")
yes, exactly. What is unclear to me is: What data does the update-by-query operation use when updating documents?
Because if the index is created by a transform, then how does the update-by-query operation know how to load the source of that document that the transform used to generate the doc?
Or is it just using the exinsting document in the destination index of the transform ?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 14, 2023, 1:05pm UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/4 "2023-11-14T13:05:40Z")
> [@pulsy](#):
>
> Or is it just using the exinsting document in the destination index of the transform ?
You would need to manually run the update\_by\_query request on the destination index, and you can set an ingest pipeline to run with this request as well.
---
### Author:  [@pulsy](https://discuss.elastic.co/u/pulsy)
#### Post date: [November 14, 2023, 1:24pm UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/5 "2023-11-14T13:24:57Z")
> You would need to manually run the update\_by\_query request on the destination index, and you can set an ingest pipeline to run with this request as well.
that's what i assumed yes. But what i still don't understand is
- The destination index is created by a transform, that is fed by some aggregations.
- When i manually run an update-by-query request on the destination index, what does the update operation actually do - which data does it use to update the matching documents?
Since i don't supply any actual values in the operation that should be set on the matching documents, is it just running the ingest pipelines on the existing document? (which is what i assume it does)
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 14, 2023, 1:55pm UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/6 "2023-11-14T13:55:36Z")
> [@pulsy](#):
>
> When i manually run an update-by-query request on the destination index, what does the update operation actually do - which data does it use to update the matching documents?
It uses the `_source` field of each document, what will be updated will depend if you will use a update query or just run an ingest pipeline with the enrich processor.
From the documentation I would assume that it would just rerun the ingest pipeline on the current data, if this will update anything will depend on the ingest pipeline and how your data looks like.
But I'm also not sure if this will impact the transform or not.
You would need to duplicate your transform and destination index to be able to test this safely.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 12, 2023, 1:55pm UTC](https://discuss.elastic.co/t/using-transforms-and-ingest-pipelines-on-data-that-changes-over-time/347101/7 "2023-12-12T13:55:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.