# Is there a soon update for transforms API

**URL:** <https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [October 1, 2021, 2:43pm UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685 "2021-10-01T14:43:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [October 1, 2021, 2:43pm UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/1 "2021-10-01T14:43:03Z")

</div>

I'm using transforms API for a use-case, where I have to run it continuously, the limitation described [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.15/transform-limitations.html) list that the max frequency I can set is every `1h`. But I'm planning to run it at night for documents indexed last 24h. is there an update coming soon or any workaround please ?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [October 6, 2021, 7:05am UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/2 "2021-10-06T07:05:46Z")

</div>

Better scheduling is planned for a future release, however I can't give you an exact date when this is coming.

> [@marone](#):
>
> ... any workaround please?

Are you using a `date_histogram` in your transform, configured with `1d` interval?

Starting with `7.15` a `date_histogram` will only process complete buckets by default, _complete_ means in this case that 24 hours must have passed. The _new_ setting is called [`align_checkpoints`](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-transform.html#put-transform-request-body). the _default_ is _true_.

With other words: Although transform triggers e.g. every hour, it will not do any work and therefore not cause any harm. It is just 1 cheap query every hour. Even if you set `frequency` to a lower value you probably won't notice. Once transforms runs the next day and once all data of the last day is available, it will create the bucket(s) of the last day.

Nevertheless we have plans for better scheduling, e.g. to specify exactly when to run a transform.

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [October 7, 2021, 8:09am UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/3 "2021-10-07T08:09:42Z")

</div>

Thank you for you reply, that's interesting! so what I understand for `7.15`, in order for my transformation to process documents ingested in the last 24 hours, I have to use a `date_histogram` so it will create a bucket for it and then execute a transformation. But my use case for my transformation doesn't use any `date_histogram`, instead I use a `pivot, group_by` and a `scripted_metric` aggs. Here is what my transformation looks like:

```auto
POST _transform/_preview
{
  "source": {
    "index": "source_index"
  },
  "dest": {
    "index": "dest_index"
  },
  "pivot": {
    "group_by": {
      "id": {
        "terms": {
          "field": "id"
        }
      }
    },
    "aggregations": {
      "latest_ts": {
        "scripted_metric": {
          "init_script": "state.timestamp_latest = 0L;",
          "map_script": """
          def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli();
          if (current_date > state.timestamp_latest && doc['host.name'].getValue() =~ /hostname2/)
          {state.timestamp_latest = current_date;}
        """,
          "combine_script": "return state",
          "reduce_script": """
          def last_doc = '';
          def timestamp_latest = 0L;
          for (s in states) {if (s.timestamp_latest > (timestamp_latest))
          {timestamp_latest = s.timestamp_latest;}}
          return timestamp_latest
        """
        }
      },
      "first_ts": {
        "scripted_metric": {
          "init_script": "state.timestamp_first = 999999999999999L;",
          "map_script": """
          def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli();
          if (current_date < state.timestamp_first && doc['host.name'].getValue() =~ /hostname1/)
          {state.timestamp_first = current_date;}
        """,
          "combine_script": "return state",
          "reduce_script": """
          def last_doc = '';
          def timestamp_first = 999999999999999L;
          for (s in states) {if (s.timestamp_first < (timestamp_first))
          {timestamp_first = s.timestamp_first;}}
          return timestamp_first
        """
        }
      },
      "time_length": {
        "bucket_script": {
          "buckets_path": {
            "min": "first_ts.value",
            "max": "latest_ts.value"
          },
          "script": """
          (params.max - params.min)/1000
           
          """
        }
      }
    }
  }
}

```

is it possible to include a date\_histogram for my transformation in order to be optimized?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [October 7, 2021, 9:36am UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/4 "2021-10-07T09:36:30Z")

</div>

> [@marone](#):
>
> so what I understand for `7.15` , in order for my transformation to process documents ingested in the last 24 hours, I have to use a `date_histogram` so it will create a bucket for it and then execute a transformation

No, you don't have to use a `date_histogram`. I assumed you are using one, because you mentioned the 24 hour requirement. The `date_histogram` in addition to your `terms` group\_by would allow you to look back in history.

Regarding your configuration:

`latest_ts` and `first_ts` return the latest and first timestamp. I wonder why you are not using `min` and `max`. This should work for calculating `time_length`, the docs contain a [similar example](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-bucket-script).

> [@marone](#):
>
> is it possible to include a date\_histogram for my transformation in order to be optimized?

I suggest to try this out, an additional `date_histogram` will create more buckets, so more documents. As said above, if you do this in addition you preserve historic values which might be useful. Note, the destination index will contain several entries by id, in order to see the latest one you can e.g. use a `top_hits` aggregation.

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [October 7, 2021, 2:33pm UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/5 "2021-10-07T14:33:39Z")

</div>

Thank you, well I didn't use `max` and `min` because I had to select `first_ts` and `latest_ts` based on a value of field `host.name`, that's why I had to do it with as above ^^

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [October 7, 2021, 3:03pm UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/6 "2021-10-07T15:03:32Z")

</div>

> [@marone](#):
>
> Thank you, well I didn't use `max` and `min` because I had to select `first_ts` and `latest_ts` based on a value of field `host.name`

Got it. It might be possible to use the `filter` aggregation and `min/max` as sub aggregation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2021, 3:04pm UTC](https://discuss.elastic.co/t/is-there-a-soon-update-for-transforms-api/285685/7 "2021-11-04T15:04:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.