# Transforms - How to check next run time
**URL:** https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150
**Category:** Elasticsearch
**Tags:** transforms
**Created:** [September 18, 2020, 5:59pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150 "2020-09-18T17:59:31Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 18, 2020, 5:59pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/1 "2020-09-18T17:59:31Z")
Hi, I created a continuous Transforms to run once a day and calculate the P95 of some fields for the previous 24hrs. When I initially created and started this Transform it created the daily P95 values from all the previous indices the Transforms was pointing to. However, it has been two days now and I haven't seen any new data from this Transforms job and written to the index. When I go click 'preview messages' inside the Transforms config I do see 24hr old data there, but for some reason it is not writing it to the index so I'm not able to see it in the Discover window. Is there a way to check when the next Transforms is scheduled to run?
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 21, 2020, 7:13am UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/2 "2020-09-21T07:13:27Z")
No, unfortunately there is no way to find out when it runs next, but you can find out when transform checked last. The [`_stats` API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-transform-stats.html) returns a field called `changes_last_detected_at`. Given that you can extrapolate the next run. The format of the field is a timestamp, with `?human=true` you can get readable format.
If there is a problem with the transform, stats should tell you.
The new data you are pushing in is not historic but "current"? Transform only checks data that is newer than the latest checkpoint according to the field you configured for `sync`.
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 22, 2020, 2:11pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/3 "2020-09-22T14:11:50Z")
When I change the frequency of the transforms to run every minute instead of once a day, I see the data stays updated. If I keep the frequency to run once a day then the data does not stay updated.
Yes, there is new data being written to the index every minute. I just feel running the transform every minute is probably putting unnecessary load on the cluster.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 22, 2020, 2:25pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/4 "2020-09-22T14:25:10Z")
Can you share your configuration (or at least the essential parts)? Are you using a `date_histogram` as `group_by`?
There should not be any difference (after 24h) in the data whether you run it with a lower frequency or not. As you said, it just puts more load on the cluster. If you run it more frequent it will constantly update and write out intermediate results.
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 22, 2020, 3:04pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/5 "2020-09-22T15:04:12Z")
This is the transforms I configured to run once a day. It is currently calculating all the historical P95 calculations, however, when I check tomorrow around the same time I will not see an update for today's P95.
```
{
"id": "transforms_p95_24hr",
"source": {
"index": [
"transforms-ptx-v3"
],
"query": {
"match_all": {}
}
},
"dest": {
"index": "transforms_p95_24hr"
},
"sync": {
"time": {
"field": "@timestamp",
"delay": "24h"
}
},
"pivot": {
"group_by": {
"@timestamp": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1d"
}
},
"a_node": {
"terms": {
"field": "a_node"
}
},
"bundle": {
"terms": {
"field": "bundle"
}
},
"site": {
"terms": {
"field": "site"
}
},
"z_node": {
"terms": {
"field": "z_node"
}
}
},
"aggregations": {
"traffic.in": {
"percentiles": {
"field": "traffic.in",
"percents": [
95
]
}
},
"traffic.out": {
"percentiles": {
"field": "traffic.out",
"percents": [
95
]
}
},
"traffic.capacity": {
"percentiles": {
"field": "traffic.capacity",
"percents": [
95
]
}
}
}
}
```
This is the one I created yesterday which runs the transform every minute for the previous 24hrs and is showing me yesterday's P95 calculations:
```
{
"id": "transforms_p95",
"source": {
"index": [
"transforms-ptx-v3"
],
"query": {
"match_all": {}
}
},
"dest": {
"index": "transforms_p95"
},
"sync": {
"time": {
"field": "@timestamp",
"delay": "60s"
}
},
"pivot": {
"group_by": {
"@timestamp": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1d"
}
},
"a_node": {
"terms": {
"field": "a_node"
}
},
"bundle": {
"terms": {
"field": "bundle"
}
},
"site": {
"terms": {
"field": "site"
}
},
"z_node": {
"terms": {
"field": "z_node"
}
}
},
"aggregations": {
"traffic.in": {
"percentiles": {
"field": "traffic.in",
"percents": [
95
]
}
},
"traffic.out": {
"percentiles": {
"field": "traffic.out",
"percents": [
95
]
}
},
"traffic.capacity": {
"percentiles": {
"field": "traffic.capacity",
"percents": [
95
]
}
}
}
}
```
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 22, 2020, 3:14pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/6 "2020-09-22T15:14:22Z")
> [@mohsin106](#):
>
> ```auto
> "sync": {
> "time": {
> "field": "@timestamp",
> "delay": "24h"
> }
>
> ```
`delay` means your data could arrive up to 24 hours late (ingest delay), this means it can take up to 24 hours between the creation of the signal, ingest, data available in elasticsearch. This does not configure how often transform runs.
I think what you are looking for is [`frequency`](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-transform.html#put-transform-request-body), `frequency` defines how often transform checks for new data. You must configure it as part of the body at the same level of `sync`, `pivot`, etc.
You do not need to re-create your transform, but you can use [update](https://www.elastic.co/guide/en/elasticsearch/reference/master/update-transform.html). In recent versions we have UI support for update.
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 22, 2020, 5:50pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/7 "2020-09-22T17:50:19Z")
> [@Hendrik\_Muhs](#):
>
> ingest delay
I believe the "delay": "24h" in the JSON output is generated by what gets entered when selecting the frequency from this WUI:
I believe the "delay" in this WUI means how often to run the transform, correct? Is there another frequency field that I should be configuring?
Since I want the daily P95 I set my Group By like this where @timestamp is "1d":
In the "Aggregation" section there is no drop down to select aggregate by 24hrs so I don't select @timestamp here, but instead, I select the IN and OUT traffic fields:
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 23, 2020, 6:08am UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/8 "2020-09-23T06:08:40Z")
> [@mohsin106](#):
>
> I believe the "delay" in this WUI means how often to run the transform, correct? Is there another frequency field that I should be configuring?
No, as I explained in the last post, delay is the ingest delay, it's the maximum time data can arrive late for various reasons. The explanation in the UI is correct.
However, it seems `frequency` misses a UI component in this wizard, I will let the UI team know.
You have 2 options: You can configure frequency by creating the transform in dev console (use "copy to clipboard", switch to dev tools, paste it there and edit it) or you use the edit function from the jobs list:
Choose edit, this should pop out:
As I don't know which version you are using, this might not be available, if so, dev console is the only option.
Thanks for the feedback, we will consider adding frequency to the UI wizard.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 23, 2020, 6:24am UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/9 "2020-09-23T06:24:54Z")
Update on the UI, in the upcoming version `7.10` `frequency` can be set on the wizard under "advanced settings", see [https://github.com/elastic/kibana/pull/77370](https://github.com/elastic/kibana/pull/77370).
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 1:49pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/10 "2020-09-23T13:49:06Z")
I'm using 7.8.1 and I do see the Frequency GUI when clicking on Edit on the Transform settings. When I checked for my 24hr Transform job the Frequency in that UI wasn't set to anything. I think I need to set the 24h time frame once while stepping through the UI when I enable "Continuous" mode, and then afterwards by clicking Edit and setting it there so that the Transform job checks for new data every 24hrs.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 23, 2020, 1:51pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/11 "2020-09-23T13:51:58Z")
> [@mohsin106](#):
>
> then afterwards by clicking Edit and setting it there so that the Transform job checks for new data every 24hrs.
Exactly, the UI in `7.8` does not support `frequency` as part of the transform creation wizard, this is unfortunate, but will be fixed with `7.10`.
Don't forget to set `delay` to a proper value.
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 1:53pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/12 "2020-09-23T13:53:14Z")
I tried setting the Frequency from the Transform setting but I can only put a max of 1h in there doesn't let me continue if I put 24h.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 23, 2020, 1:58pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/13 "2020-09-23T13:58:47Z")
Yes, that's correct, the current maximum is `1h`. Better scheduling is planned for the future. The default is `1m`.
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 2:00pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/14 "2020-09-23T14:00:42Z")
Ok so I'll set it to one hour for now, better than running the Transform job every minute. 🙂
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 2:05pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/15 "2020-09-23T14:05:08Z")
Now i see the "frequency" field in the JSON right above "sync":
```
"frequency": "1h",
"sync": {
"time": {
"field": "@timestamp",
"delay": "24h"
}
```
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 2:09pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/16 "2020-09-23T14:09:37Z")
Wait, so then when the frequency was set to its default 1m for my 24h Transform job and my delay was set to 24hrs how come I don't see data past Sept 21? I'm guessing the delay is waiting for another 24hrs to pass before allowing Transform to run again?
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 23, 2020, 2:39pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/17 "2020-09-23T14:39:33Z")
`delay` offsets the ingest delay, please have a look at the explanation in previous posts.
An example to make this more clear. With a `delay` of `24h`, transform assumes that it can _not_ query for data that is younger than `24h`, because there might still be data coming in.
Assume transform triggers at time A, it queries for data lower than `A - 24h`. That means your transform is always a day behind.
How is data ingested in your case? Where are the timestamp coming from? The default delay is `60s`, this works for most use cases and has enough wiggle room. Remember, the default refresh interval of an index in elasticsearch is `1s`, this is the minimum delay you should set.
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 5:01pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/18 "2020-09-23T17:01:50Z")
Source data is being emitted from a network device every minute and sent to a Kafka cluster. Sparc is grabbing the data from Kafka and does some calculations/manipulations on the fly before it sends that data to a specific index inside ES (IndexA). This Sparc data is being ingested every minute. Then we have a transform job that does some aggregation on the Sparc data and stores the results in a different index (IndexB).
The transform job that I'm trying to write is using data from IndexB and trying to generate daily P95 values, and then stored into a new index (IndexC). There is a template that is applied to the source data coming in and any indices generated by the transform jobs. This is how the timestamp is being mapped in the template:
```
"@timestamp": {
"type": "date"
},
```
---
### Author:  [@mohsin106](https://discuss.elastic.co/u/mohsin106)
#### Post date: [September 23, 2020, 5:05pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/19 "2020-09-23T17:05:08Z")
If I PUT the transform job from the dev console with the proper frequency of 24h, do you know if it will work with my current version of code?
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [September 23, 2020, 6:19pm UTC](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150/20 "2020-09-23T18:19:43Z")
ok, so `delay` should be the worst-case runtime for kafka, sparc, and indexing. You said that sparc runs every minute, so the default delay won't work, your delay is more than 1 minute. I do not know what kafka and sparc takes to queue and aggregate. On the other side you transform with a daily pattern, there seems no need to optimize delay, something like `5m` seems sufficient to me.
Regarding frequency, the limitation to `1h` is not a UI limitation, that means even dev console will not allow a higher value than `1h`.
[Next page](https://discuss.elastic.co/t/transforms-how-to-check-next-run-time/249150.md?page=2)