# Transform data mismatch with source index

**URL:** <https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [July 14, 2022, 2:11pm UTC](https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659 "2022-07-14T14:11:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rokcarl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rokcarl/32/53239_2.png) [@rokcarl](https://discuss.elastic.co/u/rokcarl)\
**Post date:** [July 14, 2022, 2:11pm UTC](https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659/1 "2022-07-14T14:11:19Z")

</div>

Hi,

I'm using transforms to aggregate our access logs to a daily aggregate, e.g. "customer A had 500 requests totalling 1000 credits".

**Problem**  
The problem that I now have is that I want to verify that the transformed data adds up, i.e. is the count of requests the same? So I do the request on both indices, the results are really close, but not the same:

 ![Screenshot 2022-07-14 at 16.00.10](https://us1.discourse-cdn.com/elastic/original/3X/9/7/972860bb640bb7d143c2c39c6650ae427ca7fc1d.png)

**My intuition**  
The differences (last column) are so low that I suspect that there's some timing issue, but neither timezones nor transformation lag make sense to me.

**Sources & code**  
Here's my [transform code](https://gist.github.com/rokcarl/b1ef9eb402f3a43bdcff573c926ca335#file-gistfile1-txt-L41), the queries for the [normal](https://gist.github.com/rokcarl/6a5db307a9dd3dac2325cf7a20165648) index and for the [transform](https://gist.github.com/rokcarl/467b506a1ea908d159e6c20a8925ca77) index, basically identical except for the normal one needed an additional aggregation to sum the requests which the other already has through the transform and change of the field name.

Any idea what might be going on?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 14, 2022, 3:26pm UTC](https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659/2 "2022-07-14T15:26:56Z")

</div>

As part of your queries I see you define a timezone. But you don't define that timezone in the transform. There you pre-aggregate with a `date_histogram`, so bucketing happens there. That means date bucketing is already done in the transformed index. I think you should define the timezone in the transform or configure the `date_histogram` with more granularity, e.g. `1h`. That way your query on the transformed index can adjust the buckets. Now you basically already lost the precision after the transform.

Another reason for the mismatch might be the terms grouping. Can you verify that the `customer` field is never `null`? Transform by default ignores it otherwise or you set `missing_bucket` to `true`.

---

<div class="post-metadata">

**Author:** ![rokcarl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rokcarl/32/53239_2.png) [@rokcarl](https://discuss.elastic.co/u/rokcarl)\
**Post date:** [July 15, 2022, 7:07pm UTC](https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659/3 "2022-07-15T19:07:23Z")

</div>

You were spot-on. I set the granularity to `1h` and filtered to where `customer` is set.  
Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2022, 7:07pm UTC](https://discuss.elastic.co/t/transform-data-mismatch-with-source-index/309659/4 "2022-08-12T19:07:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.