# Transform missing data **URL:** \ **Category:** Elasticsearch\ **Tags:** transforms\ **Created:** [July 4, 2022, 1:56pm UTC](https://discuss.elastic.co/t/transform-missing-data/308818 "2022-07-04T13:56:22Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rokcarl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rokcarl/32/53239_2.png) [@rokcarl](https://discuss.elastic.co/u/rokcarl)\ **Post date:** [July 4, 2022, 1:56pm UTC](https://discuss.elastic.co/t/transform-missing-data/308818/1 "2022-07-04T13:56:22Z")
I have a transform that aggregates my stats index data for long-term storage, but the results are no longer correct. My stats index stores each request (one document = one request) and stores the credits used by that request, e.g. 6. I transform this into one-day aggregations, e.g. customer A did 1200 requests on Monday with a total credits 2400. The aggregations for the transform are: - credits: `{ "sum": { "field": "credits" } }`, - requests: `{ "value_count": { "field": "timestamp" }}`. I run this with a frequency of 1h and this sync: `{"time": {"field": "timestamp","delay": "90m"}}` and I group by a 1d fixed interval and by customer. Here's the [entire transform json](https://gist.github.com/rokcarl/bbb5577a74ccd733de5652c898aa4987), you can ignore the scripted fields. When I then graph the count of records and the sum of credits, neither graph is the same between a normal index and the transform. Why is that? ![Screenshot 2022-07-04 at 15.45.26](https://us1.discourse-cdn.com/elastic/original/3X/c/c/cc7e89f36273200dd8c4a4e445fcb1ad8f87a81e.png) ☝ left is real graph, right is transform. Do I need to run the transform at the end of the day? As far as I'm aware that's not needed? ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 5, 2022, 8:02am UTC](https://discuss.elastic.co/t/transform-missing-data/308818/2 "2022-07-05T08:02:20Z")
You specified a query filter that limits transform to only access data from the last 3 hours, actually even just 1.5 hours, because of the 1.5 hour delay: ```auto "must": [{ "range" : {"timestamp": {"gte" : "now-3h"}} }] ``` Such a filter is never a good idea. Transform takes care of query filters themselves. You find an explaination about checkointing in the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-checkpoints.html). Such a query is only useful if you want to set a fixed start date, e.g. if you want to prevent to transform all data from the past. > [@rokcarl](#): > > Do I need to run the transform at the end of the day? Which version are you using? If relatively recent, I suggest to read about [`settings.align_checkpoints`](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html#put-transform-query-parms). This setting has a default value of `true`, it causes the transform to wait until a bucket is complete before it creates it. In your case of 1 day buckets that means it waits until midnight plus your 1.5 hour delay, so roughly creates new data between 1:30 - 2:30 in the morning. If you prefer to see intermediate results during the day you can set `align_checkpoints` to `false`. In this case the transform will update the results during the day. As this means more work, it comes for the price of performance. > TL/DR > If you use a relatively recent version and `align_checkpoints` is set to `true` your query from above causes data problems, because transform is only able to get data from at most 1.5 hours of the last day. Regarding your unusually large query delay: What's the reason to configure a 1.5 hour delay? As alternative you could use an [ingest timestamp](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-checkpoints.html#sync-field-ingest-timestamp) instead. ---
**Author:** ![rokcarl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rokcarl/32/53239_2.png) [@rokcarl](https://discuss.elastic.co/u/rokcarl)\ **Post date:** [July 14, 2022, 2:12pm UTC](https://discuss.elastic.co/t/transform-missing-data/308818/3 "2022-07-14T14:12:08Z")
Thank you, that was it and it fixed my problem. I'm facing a new one though about data mismatch, but I opened a [new thread](https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659) for that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 11, 2022, 2:12pm UTC](https://discuss.elastic.co/t/transform-missing-data/308818/4 "2022-08-11T14:12:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.