# Can some one explain how the transform runs it in this scenario

**URL:** <https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888>\
**Category:** Kibana\
**Created:** [June 26, 2020, 5:20pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888 "2020-06-26T17:20:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![anjilinga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anjilinga/32/86010_2.png) [@anjilinga](https://discuss.elastic.co/u/anjilinga)\
**Post date:** [June 26, 2020, 5:20pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888/1 "2020-06-26T17:20:03Z")

</div>

Hi,

I am using transform index, and i am grouping the records by day wise to find out max.

but the frequency of the transform can be set maximum as 1hour.

if the transform runs at next hour will it consider the records already processed to group by day.

Example.  
Time field value  
8:10 - 10  
8:35 - 25  
8:50- 20

transform runs at 9, and displayed max as 25

9:20 - 20  
9:50- 10

transform runs at 10, will it consider the previous records and display as 25

10:25 - 30  
10:55 - 20

ttransform runs at 11, will it consider the previous records and display as 30

if delay field is 60s, it is not processing records, what is the exact significance of this field in sync object ?

---

<div class="post-metadata">

**Author:** ![mykael](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mykael](https://discuss.elastic.co/u/mykael)\
**Post date:** [June 27, 2020, 2:20pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888/2 "2020-06-27T14:20:14Z")

</div>

The sync field is used with the high water mark and the delay to determine which records in the input index it is going to do the calculation for. Delay may need to be \>= frequency to ensure it processes all records. It may be querying for sync \> (now - delay) every frequency interval - sync \> (now - 60s) every hour - in your case, which would miss 59 minutes worth of data.

What I think happens when it runs is that it groups all of the input records - the date histogram for the @timestamp works by whole calendar days. So it works out that all of the input records are on the same day. Then it does a query against the index for all records that occurred on that day, groups them and writes the resulting documents to the output index, creating new versions (or maybe maybe updating) those documents as it goes.

---

<div class="post-metadata">

**Author:** ![anjilinga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anjilinga/32/86010_2.png) [@anjilinga](https://discuss.elastic.co/u/anjilinga)\
**Post date:** [June 27, 2020, 3:09pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888/3 "2020-06-27T15:09:09Z")

</div>

Thanks Clarke,

if the frequency is 1 hr. when it runs transform at 10. it consider the records greater than 9:59.  
it will loose 9 to 9:59 data ? is my assumption is correct ?  
when it runs at 10 or 11 or 12, it should consider previous records from the source index. in this time will it consider the missing data ?  
if sync field is not @timestamp, then what is the formaula for delay value( if the sync field is another time field and it is having 30 min difference to present time)  
delay \>= frequency + 30 min ?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [June 29, 2020, 8:15pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888/4 "2020-06-29T20:15:56Z")

</div>

`frequency` defines how often transform looks for new data _and_ in case of a failure how quick it re-tries. This setting only defines scheduling, this setting has _no_ impact on how the data is transformed. With other words, using different `frequencies` does not lead to different data.

`sync` and the sub-setting `delay` does not impact how data is transformed, assuming you set it up correctly: `delay` defines the ingest delay, it means: "When is it safe to query for data?". The time used for the timestamp field can have different delays, e.g. if you feed the timestamp from an external system, it might be, that you batch data and send it e.g. every 5 minutes. For this case `delay` must be 5 minutes _plus_ whatever it takes to transfer the data over the network and index it in elasticsearch (`refresh_interval`).

A continuous transform works in 2 steps:

- identify the data points that need to be updated
- re-create the data points that needed to be updated.

If you configure `sync` with a `delay` to low, step 1 might miss data points to be updated.

Regarding your example:

When it runs the transform at 10, assuming it run it the last time at 9.

1. query the `source` between 9 and 10 and e.g. identify that `a` and `b` have been changed (but not `c` and `d`).
2. query the `source` _till_ 10 filtered by `a` and `b` and update the documents for `a` and `b`.

Note:

- in step 2 it queries _all_ data
- `a` and `b` can be terms but also ranges if you are grouping by `date_histogram`
- if the query runs at 10 it does _not_ query "lower than 10" but "lower than 10 - delay", accordingly the range is step 1 is `9-delay <= x < 10-delay`

So again both `sync` and `frequency` do not affect _how_ data is transformed but only _how_ and _when_ it is updated. The transformation is defined in your `group_by`. If you are not getting the expected it results, you might have a problem with setting up `sync` or a problem in your data.

I think I can better help you if you post your config and explain what you expect.

---

<div class="post-metadata">

**Author:** ![anjilinga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anjilinga/32/86010_2.png) [@anjilinga](https://discuss.elastic.co/u/anjilinga)\
**Post date:** [July 6, 2020, 5:26pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888/5 "2020-07-06T17:26:11Z")

</div>

Thanks hendrick. i got the idea now

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 3, 2020, 5:26pm UTC](https://discuss.elastic.co/t/can-some-one-explain-how-the-transform-runs-it-in-this-scenario/238888/6 "2020-08-03T17:26:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.