X-Pack Watcher - Trigger time drift gap

[Steve_Stevens]

Afternoon all!

I'm using X-Pack Watchers in Kibana to poll the logs every 1 minute for particular faults.
The watcher itself is set to trigger every "1m":

and to query (amongst other things) for logs between 1m of the scheduled time:

This is working well, however, I have noticed that the trigger time is actually slightly drifting by ~100ms between every execution.

We've seen already some logs being missed as part of our alert due to falling between this 100ms gap.
Does anyone have any suggestions for me to improve the trigger time so that the gap is closed?

Thanks in advance!
Steve

[Steve_Stevens]

I tried a couple of alternate approaches but it appears that "1m" uses last triggered time as the start of its next scheduled operation. This would explain the drift (though that is speculation).

Instead, I tried a cron based schedule:
"trigger": {
"schedule": {
"cron": "0 */1 * * * ?"
}
},

which, when you look at the trigger_event, seems to force the scheduled time to be on-the-dot:
"trigger_event": {
"type": "schedule",
"triggered_time": "2019-01-28T12:14:00.209Z",
"schedule": {
"scheduled_time": "2019-01-28T12:14:00.000Z"
}
},

Whilst the execution still happens "ms" after the schedule time, the query is now at least scheduling based on the exact minute.