# Transform with fixed time interval and different sync delay

**URL:** <https://discuss.elastic.co/t/transform-with-fixed-time-interval-and-different-sync-delay/308075>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [June 24, 2022, 4:26am UTC](https://discuss.elastic.co/t/transform-with-fixed-time-interval-and-different-sync-delay/308075 "2022-06-24T04:26:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Avarjana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avarjana/32/104822_2.png) [@Avarjana](https://discuss.elastic.co/u/Avarjana)\
**Post date:** [June 24, 2022, 4:26am UTC](https://discuss.elastic.co/t/transform-with-fixed-time-interval-and-different-sync-delay/308075/1 "2022-06-24T04:26:11Z")

</div>

I have a transform to identify anomaly and trigger alert in one of my index. Following are the details of the behavior.

**Expected behavior**

- For each **minute** , aggregate **last 5 minutes** data and check for alerts. ❎
- Send records to relevant indexes. ✅

**Problematic current behavior**

- For each **5 minutes** , aggregate the **last 5 minutes** data and check for alerts.
- Send alerts to relevant indexes.

**Cause of the problem**  
Transform rule has fixed time interval of 5 minutes for `group_by` and sync delay is `60s (1 min)`.

```auto
"pivot": {       
       "group_by": {
           …
           "@timestamp": {
               "date_histogram": {
                   "field": "@timestamp",
                   "fixed_interval": "5m"
               }
           }
       },
      …

```

**This will capture events for**

- XX:00 - XX:05
- XX:05 - XX:10
- XX:10 - XX:15
- ....

This interval can be changed to 1 minute but it will check alerts in last one minute record per minute.

**Issue**  
Alerts get only triggered **per five minutes** even if the event happens at the first minute of the interval.

It would be highly appreciated if you can provide a suggestion for this problem.

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [June 24, 2022, 7:10am UTC](https://discuss.elastic.co/t/transform-with-fixed-time-interval-and-different-sync-delay/308075/2 "2022-06-24T07:10:59Z")

</div>

Per default transform creates buckets after the bucket is complete, that's it waits 5 minutes, if your `date_histogram` is configured with 5 minutes interval. This improves performance, because transform does not need to update documents. You can change this using the setting `align_checkpoints`. It is default `true` and can be set to `false`. This will tell transform to process incomplete buckets for the price of more updates and therefore some performance penalty. You find this setting in the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-transform.html#put-transform-request-body)

Note that you will still have a waiting time of at least 1 minute if your `sync` delay is set to `60s`, because transform will only query for data that is at least 1 minute old. This setting compensates ingest delays and data coming in in different order. If you know that your configured timestamp is guaranteed to reach elasticsearch earlier, you can decrease this setting to further optimize the time to trigger the alert. An even better approach which will compensate any problem on the data ingestion is the use of an ingest timestamp as explained [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-checkpoints.html#sync-field-ingest-timestamp). By using an ingest timestamp you can decrease the setting for `sync` to e.g. `5s` (You can't decrease it to `0s`, because the refresh interval of a lucene index per default is `1s`, so I think `2s` should be the minimum).

---

<div class="post-metadata">

**Author:** ![Avarjana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avarjana/32/104822_2.png) [@Avarjana](https://discuss.elastic.co/u/Avarjana)\
**Post date:** [June 27, 2022, 11:37am UTC](https://discuss.elastic.co/t/transform-with-fixed-time-interval-and-different-sync-delay/308075/3 "2022-06-27T11:37:51Z")

</div>

Thank you for this great explanation. Highly appreciate the support. 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2022, 11:38am UTC](https://discuss.elastic.co/t/transform-with-fixed-time-interval-and-different-sync-delay/308075/4 "2022-07-25T11:38:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.