Hi, I am running a continuous transform at a 15-minute frequency
to transform the data from index_A
to index_B
.
In the transform, I have set a 120s sync delay
time to avoid missing the latest data. However, in the aggregation scripts, I have a function that would require finding out the time elapsed between the document @timestamp
and the trigger time of the transformation.
That is, suppose the continuous transformation triggers at a time T
, I would like to get the time T - doc[@timestamp].value
in the Scripted metric aggregation i.e. init_script
/ map_script
/ reduce_script
My questions are:
-
As I cannot find a way to get the transformation triggers time
T
, I am currently using theSystem.currentTimeMillis()
as the timeT
to get the time elapsed, however, I am concerned that the possiblesync delay
time will cause a difference in the script. -
Is there any way to get the transformation triggers time
T
inside the Scripted metric aggregation? I have seen that thetime_upper_bound_millis
from the transform page, is this time ``time_upper_bound_millis` equal to the transformation triggers time I need? If it is, is there any way to get it in the Scripted metric aggregation?
Attached is the transform information.
{
"id": "transform script",
"version": "7.14.2",
"create_time": -,
"source": {
"index": [
"index_A"
],
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-15m"
}
}
}
]
}
}
},
"dest": {
"index": "index_B",
"pipeline": ""
},
"frequency": "15m",
"sync": {
"time": {
"field": "@timestamp",
"delay": "120s"
}
},
"pivot": {
"group_by": {
}
},
"aggregations": {
"@timestamp": {
"max": {
"field": "@timestamp"
}
},
"script_field": {
"scripted_metric": {
"combine_script": "return state",
"init_script": {
"id": "some_init_script"
},
"map_script": {
"id": "some_map_script"
},
"params": {
},
"reduce_script": {
"id": "some_reduce_script"
}
}
}
}
},
"description": "DEMO",
"settings": {
"max_page_search_size": 10000
}
}