# ES transforms behaviour change after upgrading from 7.10.0 to 7.16.1

**URL:** <https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [December 22, 2021, 2:48pm UTC](https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700 "2021-12-22T14:48:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![brna](https://avatars.discourse-cdn.com/v4/letter/b/edb3f5/32.png) [@brna](https://discuss.elastic.co/u/brna)\
**Post date:** [December 22, 2021, 2:48pm UTC](https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700/1 "2021-12-22T14:48:03Z")

</div>

I am observing a change in the way es transforms work after the cluster upgrade from 7.10.0 to 7.16.1  
It is a continuous transform with filter on source index that would get all of the documents from the source index with yesterday's and today's timestamps:

```auto
"range": {
              "@timestamp": {
                "gte": "now-1d/d",
                "lt": "now"
              }
            }

```

Transform then does some aggregations and does grouping on timestamp field (with 1 day interval):

```auto
"pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1d"
        }
      }
[..]

```

Before upgrade, this would create a 2 documents - one for today, one for yesterday.  
After the upgrade, only document for yesterday is created and document for today has to wait until the day passes and it becomes "yesterday" document, and then gets created in the destination index.

Is this a bug or a desired change of transform behaviour in the newer version?

Thanks

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [December 22, 2021, 3:01pm UTC](https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700/2 "2021-12-22T15:01:14Z")

</div>

We indeed changed the default behavior.

Did you re-created the transform or did you upgraded the cluster with an _existing_ transform?

Existing transform should _not_ change behavior, new ones align buckets on default for _new_ created transforms. You can change this in the config using:

```auto
{
 "settings": {
    "align_checkpoints": false
    }
}

```

See the `settings` section in the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-transform.html). Note, `settings` can be changed via `_update` on existing transforms. You don't need to re-create the transform.

---

<div class="post-metadata">

**Author:** ![brna](https://avatars.discourse-cdn.com/v4/letter/b/edb3f5/32.png) [@brna](https://discuss.elastic.co/u/brna)\
**Post date:** [December 22, 2021, 4:35pm UTC](https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700/3 "2021-12-22T16:35:15Z")

</div>

Thanks for the quick response! I managed to fix my transforms and destination index looks as expected.

Worth noting:  
My transforms were created more than 2 months ago, so they were existing and still got the behaviour changed after the upgrade.  
I needed to restart transforms after \_update in order to have todays documents in the destination index.

Can you please confirm whether this [PR](https://github.com/elastic/elasticsearch/pull/74004) introduced the change in the behaviour (7.15.0)

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [December 22, 2021, 10:05pm UTC](https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700/4 "2021-12-22T22:05:38Z")

</div>

Great and thanks for the answer!

Yes, we introduced this change in [74004](https://github.com/elastic/elasticsearch/pull/74004). However we noticed your problem already and fixed your regression with [81729](https://github.com/elastic/elasticsearch/pull/81729).

That means if you would have upgraded to a recent version, e.g. `7.16.2`, your old transforms would work as before. The problem only appears in the `7.15.x`.

I am sorry for the inconvenience.

The motivation of the change: most users do not care about interim bucket results, but only want complete buckets. Writing interim buckets mean more writes, more updates and therefore more resources. That's why we decided that it is of interest for the majority to change it and save resources. Of course existing transforms should not have changed, that was a regression and we fixed this in [81729](https://github.com/elastic/elasticsearch/pull/81729). And for those that like to have interim results - like you - we introduced the setting.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2022, 10:06pm UTC](https://discuss.elastic.co/t/es-transforms-behaviour-change-after-upgrading-from-7-10-0-to-7-16-1/292700/5 "2022-01-19T22:06:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.