ES transforms behaviour change after upgrading from 7.10.0 to 7.16.1

[brna]

I am observing a change in the way es transforms work after the cluster upgrade from 7.10.0 to 7.16.1
It is a continuous transform with filter on source index that would get all of the documents from the source index with yesterday's and today's timestamps:

"range": {
              "@timestamp": {
                "gte": "now-1d/d",
                "lt": "now"
              }
            }

Transform then does some aggregations and does grouping on timestamp field (with 1 day interval):

"pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1d"
        }
      }
[..]

Before upgrade, this would create a 2 documents - one for today, one for yesterday.
After the upgrade, only document for yesterday is created and document for today has to wait until the day passes and it becomes "yesterday" document, and then gets created in the destination index.

Is this a bug or a desired change of transform behaviour in the newer version?

Thanks

[Hendrik_Muhs]

We indeed changed the default behavior.

Did you re-created the transform or did you upgraded the cluster with an existing transform?

Existing transform should not change behavior, new ones align buckets on default for new created transforms. You can change this in the config using:

{
 "settings": {
    "align_checkpoints": false
    }
}

See the settings section in the docs. Note, settings can be changed via _update on existing transforms. You don't need to re-create the transform.

[brna]

Thanks for the quick response! I managed to fix my transforms and destination index looks as expected.

Worth noting:
My transforms were created more than 2 months ago, so they were existing and still got the behaviour changed after the upgrade.
I needed to restart transforms after _update in order to have todays documents in the destination index.

Can you please confirm whether this PR introduced the change in the behaviour (7.15.0)

[Hendrik_Muhs]

Great and thanks for the answer!

Yes, we introduced this change in 74004. However we noticed your problem already and fixed your regression with 81729.

That means if you would have upgraded to a recent version, e.g. 7.16.2, your old transforms would work as before. The problem only appears in the 7.15.x.

I am sorry for the inconvenience.

The motivation of the change: most users do not care about interim bucket results, but only want complete buckets. Writing interim buckets mean more writes, more updates and therefore more resources. That's why we decided that it is of interest for the majority to change it and save resources. Of course existing transforms should not have changed, that was a regression and we fixed this in 81729. And for those that like to have interim results - like you - we introduced the setting.