Watcher: "daily" schedule doesn't work

I am running Elastic 6.2.2. I have a Watcher query that works great as long as I use an "interval" as the trigger schedule, but does not fire if I use a "daily" specifier. The problem occurred when going from this:

"trigger": {
"schedule": {
"interval": "5m"
}

to this:

"trigger": {
"schedule": {
"daily" : { "at" : "13:25" }
}

Initially, I had the watch triggering every 5 minutes for testing, then I wanted to trigger it just once per day. When I first saved the watch with modified trigger, it kept on firing every 5 minutes! Then I deleted that watch entirely and created a new one with the same code. The watch doesn't fire at the daily time indicated. It does execute just fine when run manually or via 'Simulate'. This looks like a Watcher bug -- am I missing something?
Thanks
Nathan

Hey,

I just tested the following watch and it worked (note that the time is in UTC timezone).

PUT _xpack/watcher/watch/my_watch
{
  "trigger": {
    "schedule": {
      "daily" : {
        "at" : "07:18"
      }
    }
  },
  "input": {
    "simple" : {
      "foo" : "bar"
    }
  },
  "actions": {
    "logme": {
      "logging": {
        "text": "{{ctx.payload}}"
      }
    }
  }
}

I created this watch via kibana dev-tools. Can you maybe explain the exact steps you did to create this watch? Did you use the dev-tools or something else?

Can you also share the output of the GET Watch API, as well as a search against the watches index

GET .watches/_search
{
  "query" : {
     "term" : { "_id" : "YOUR_WATCH_ID_HERE" }
  }
}

Thank you!

--Alex

Thanks Alex,
the watch is one that I built mostly in dev-tools based on other examples of watches, including some that we are currently using in production. However, all of our other successful watches use "interval" for scheduling.
Here's the search for my test watch: it was configured to run at 2:30 pm today, and of course 2:30 came and went without a trigger. The watch runs fine in simulation and if manually triggered.

GET .watches/_search
{
"query" : {
"term" : { "_id" : "PrismWatch_A1C_YNH_Test"}
}
}
gives us

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": ".watches-6",
        "_type": "doc",
        "_id": "PrismWatch_A1C_YNH_Test",
        "_score": 1,
        "_source": {
          "trigger": {
            "schedule": {
              "daily": {
                "at": [
                  "14:30"
                ]
              }
            }
          },
          "input": {
            "search": {
              "request": {
                "search_type": "query_then_fetch",
                "indices": [
                  "<prism*>"
                ],
                "types": [],
                "body": {
                  "size": 1000,
                  "query": {
                    "bool": {
                      "must": [
                        {
                          "match": {
                            "lrr": "13943"
                          }
                        },
                        {
                          "regexp": {
                            "pt_facility.keyword": "{{ctx.metadata.facility}}.*"
                          }
                        },
                        {
                          "script": {
                            "script": {
                              "source": "if (!doc['pt_age_at_collect.years.keyword'].empty) {int x = Integer.parseInt(doc['pt_age_at_collect.years.keyword'].value); (x >=18) && (x <= 70)}",
                              "lang": "painless"
                            }
                          }
                        },
                        {
                          "script": {
                            "script": {
                              "source": "float x = Float.parseFloat(doc['result.keyword'].value); (x > 6.5) || (x <5.7)",
                              "lang": "painless"
                            }
                          }
                        },
                        {
                          "range": {
                            "received_ts_utc": {
                              "gte": "now-{{ctx.metadata.window_hours}}h"
                            }
                          }
                        }
                      ]
                    }
                  }
                }
              }
            }
          },
          "condition": {
            "script": {
              "source": "return ctx.payload.hits.total > 0;",
              "lang": "painless"
            }
          },
          "transform": {
            "script": {
              "source": "ctx.payload.hits.hits.stream().map(it -> ['spec_id':it._source.order_id,'result':it._source.result,'cohort': (Float.parseFloat(it._source.result) > 6.5) ? 'Diabetic' : 'Normal', 'facility':it._source.pt_facility.substring(0,3), 'spec_recv_time': it._source.spec_recv_ts.replace('T',' ')]).collect(Collectors.toList());",
              "lang": "painless"
            }
          },
          "actions": {
            "my-logging-action": {
              "logging": {
                "level": "info",
                "text": """
The following {{ctx.metadata.facility}} outpatient (age 18-70 years) specimens with normal or diabetic A1C results were received in the last {{ctx.metadata.window_hours}} hours:
spec_id	result	spec_recv_time	cohort
{{#ctx.payload._value}}{{spec_id}}	{{result}}	{{spec_recv_time}}	{{cohort}}

{{/ctx.payload._value}}
({{ctx.metadata.watch_rev}})
"""
              }
            },
            "email_alert": {
              "email": {
                "profile": "standard",
                "to": [
                  "(Email List Redacted)"
                ],
                "subject": "Watcher Alert - Prism A1C Monitoring-{{ctx.metadata.facility}}",
                "body": {
                  "text": """
The following {{ctx.metadata.facility}} outpatient (age 18-70 years) specimens with normal or diabetic A1C results were received in the last {{ctx.metadata.window_hours}} hours:
spec_id	result	spec_recv_time	cohort
{{#ctx.payload._value}}{{spec_id}}	{{result}}	{{spec_recv_time}}	{{cohort}}

{{/ctx.payload._value}}
({{ctx.metadata.watch_rev}})
"""
                }
              }
            }
          },
          "metadata": {
            "name": "PrismWatch_A1C_YNH_Test",
            "watch_rev": "PrismWatch_A1C_YNH v3.0 2018-08-09",
            "facility": "YNH",
            "window_hours": 24,
            "xpack": {
              "type": "json"
            }
          },
          "status": {
            "state": {
              "active": true,
              "timestamp": "2018-08-10T18:18:37.501Z"
            },
            "actions": {
              "my-logging-action": {
                "ack": {
                  "timestamp": "2018-08-10T18:18:37.501Z",
                  "state": "awaits_successful_execution"
                }
              },
              "email_alert": {
                "ack": {
                  "timestamp": "2018-08-10T18:18:37.501Z",
                  "state": "awaits_successful_execution"
                }
              }
            },
            "headers": {
              "_xpack_security_authentication": "y7nvAgAIcHJpY2VuYXYBD2hlbGl4X2RzX2FkbWlucwoCB2xkYXBfZG4AQUNOPXByaWNlbmF2LE9VPVVzZXJ...many kb of authentication token...BGxkYXAA"
            },
            "version": -1
          }
        }
      }
    ]
  }
}

Querying for all watches returns info on 14 watches altogether. Do you also want to see the full output of the wider search?
GET .watches/_search
{
"query": {
"match_all": {}
}
}

Thanks,
Nate

More possibly relevant info:
I also tried scheduling using a "cron" expression, which also didn't work.
I note that Watcher uses the Quartz job scheduler; is it possible that the "daily", "monthly" expressions also rely on Quartz and that we somehow don't have Quartz installed as part of our X-Pack installation?
Thanks much
Nate

quartz is not used as a dependency, there is just one class file copied from it, so no need to worry about that.

Can you also share the watch history please? Thanks!

Hi Alex -
Sorry for the delayed reply, just got back from vacation. I have just realized that the only problem here is that I have not been taking into account the difference between UTC and local time!
I set up a test watch to run daily at 12:02 as shown below. I have been expecting this to run at two minutes past noon, but it runs at 8:02 am Eastern time. All my other scheduled watches that I've tested appeared to not work because they were set to execute four hours earlier than I expected.
So I just need to adjust my schedule to account for UTC time and all is well.
Thanks for your help and sorry for the wild goose chase!
Best
Nathan

  "trigger": {
    "schedule": {
      "daily": {
        "at": [
          "12:02"
        ]
      }
    }
  },
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.