# Questions Regarding Deleting Source Data After Transform
**URL:** https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353
**Category:** Elasticsearch
**Tags:** transforms
**Created:** [July 22, 2021, 8:02am UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353 "2021-07-22T08:02:03Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Ashfur](https://discuss.elastic.co/u/Ashfur)
#### Post date: [July 22, 2021, 8:02am UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/1 "2021-07-22T08:02:03Z")
From what I read from the other posts, if we are running a continuous transform, we should not delete any data from the source index as that will trigger the transform to re-compute and the deleted records will no longer be taken into account in the summarized index after refresh.
I have a few questions on that topic:
1. Does this apply to both pivot and latest transform?
2. If we are only interested in the summarized view and not the raw records in the source index, is there any other things we can do to reduce the storage cost from the source index if we cannot delete them? Writes into the source index can be very heavy.
Thanks.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [August 4, 2021, 2:17pm UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/2 "2021-08-04T14:17:01Z")
> [@Ashfur](#):
>
> - Does this apply to both pivot and latest transform?
This applies to pivot, however if you use a date\_histogram in pivot, you don't need to keep old data from old buckets.
Let me give you an example: Assume you pivot on user data e.g. `user_id` and you calculate values based on the data you have for that user. If new data comes in for this `user_id`, the values for that user will be recalculated. If you meanwhile deleted data, the values might be wrong.
However, in some cases this might be what you want, e.g. if you want to transform data to get a view of the last X days.
With other words: if you care about a global view, without aging out, don't delete. If not, it might be ok to delete. Another example, if instead of `user_id` you pivot on `order_id` and an order is _not_ changed after a certain amount of time, you can delete the source for it, because you won't recalculate the data for that order anymore.
The deletion of the source data does _not_ trigger a recalculation, only adding new data with a field you pivot on does. For this case you need older data.
Coming back to the date\_histogram: If you group daily and delete data that is older than a day, this isn't a problem, because transform would _not_ recalculate data for that bucket. This is similar to the `order_id` example: If there is no trigger for recalculation you can safely delete the source for it.
> [@Ashfur](#):
>
> If we are only interested in the summarized view and not the raw records in the source index, is there any other things we can do to reduce the storage cost from the source index if we cannot delete them? Writes into the source index can be very heavy.
I suggest to have a look at rollup, too. Transform is fine, but the intention of rollup is the data compaction/reduction use case.
---
### Author:  [@Ashfur](https://discuss.elastic.co/u/Ashfur)
#### Post date: [August 4, 2021, 8:47pm UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/3 "2021-08-04T20:47:39Z")
Hi Hendrik. Thanks for the context. In some situations, we do want a global view of pivot like the user\_id, so transform may not work for us in those cases. I have looked into rollup as well, but it doesn't support some of the aggregations we need like cardinality.
It seems we have to stick with keeping old data for those use cases (global/cardinality), but we will consider rollup or data histogram in other places.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [August 5, 2021, 6:16am UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/4 "2021-08-05T06:16:05Z")
To reduce the amount of data you have to keep you can consider stacking transforms together. E.g. to compact the data you could build daily/weekly or monthly summaries. On top of the output of such a transform you could run another transform that creates the final pivot you need.
With other words: The output of a transform is an index, therefore you can run another transform on this index. For cardinality this should work, for things like `avg` we have a `weighted_avg` aggregation to build averages on averages correctly.
---
### Author:  [@Ashfur](https://discuss.elastic.co/u/Ashfur)
#### Post date: [August 5, 2021, 10:57pm UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/5 "2021-08-05T22:57:02Z")
Thanks. The transform over transform seems like an interesting solution.
Using an example where the raw table contains a record of each item purchase (user\_id, item, timestamp) with cardinality.
We first run a continuous transform to convert the raw data into a summary of "the set of item each user has bought each day" - (user\_id, date, item\_sets)
We can then run another continuous transform on top of that to get "the set of item each user has bought since the beginning of time" - (user\_id, item\_sets)
This way, we can safely delete old data in the raw table since we won't have new records getting inserted with an old timestamp, so we won't trigger a recompute.
Is this the correct understanding?
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [August 6, 2021, 2:23pm UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/6 "2021-08-06T14:23:42Z")
yes, that's right.
Note that for the 2nd continuous transform you need a timestamp field to `sync` on. You can use the timestamp from the daily bucket, however in this case the `delay` for the 2nd transform needs to be `24h` + the delay of the 1st transform. If that's to slow for you, you can either configure the 1st transform with a lower interval or you add an ingest timestamp using an ingest pipeline.
See: [Dec 12th, 2018: [EN][Elasticsearch] Automatically adding a timestamp to documents](https://discuss.elastic.co/t/dec-12th-2018-en-elasticsearch-automatically-adding-a-timestamp-to-documents/159314)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 3, 2021, 2:24pm UTC](https://discuss.elastic.co/t/questions-regarding-deleting-source-data-after-transform/279353/7 "2021-09-03T14:24:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.