Watcher stopped triggering

Elastic Stack Kibana
1

Hi,

I put up two Kibana Watchers in two different deployments last week. Now, one of the watchers doesn't trigger. It randomly stopped last friday, even though I didn't edit anything around that time. Now, even if I delete the watcher, create a new one, etc.. no other watcher works. No trigger, i.e, nothing shows up in execution history. My intervall is 300s.

What may be the issue? The code is almost identical to the watch that works well in the other deployment.

Here is the code:

{
  "trigger": {
    "schedule": {
      "interval": "300s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "functionbeat*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 10,
          "query": {
            "bool": {
              "must": {
                "bool": {
                  "should": {
                    "terms": {
                      "level": [
                        "critical",
                        "alert",
                        "emergency"
                      ]
                    }
                  }
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "{{ctx.trigger.scheduled_time}}||-315s",
                    "lte": "{{ctx.trigger.scheduled_time}}",
                    "format": "strict_date_optional_time||epoch_millis"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "if (ctx.payload.hits.total > 0) return true;else return false;",
      "lang": "painless"
    }
  },
  "actions": {
    "slack_1": {
      "slack": {
        "message": {
          "text": """[{{ctx.metadata.name}}] just noticed following exceptions:

{{#ctx.payload.hits.hits}}- [{{_source.application}}] {{_source.level}}({{_source.app.monolog_level}}) with message:
    [{{_source.message}}]
    [Link="random url"/{{_index}}?id={{_id}}]


{{/ctx.payload.hits.hits}} in the last 5 minutes!"""
        }
      }
    }
  }
}
2

This happened after upgrading to v8.1.2

3

It works well to manually trigger the watcher via API, so i have no clue why it isn't triggering automatically

4

When i run GET _watcher/stats

This is returned:

"manually_stopped" : false,
  "stats" : [
    {
      "node_id" : "SnOONmkeRdagszfnBE40pg",
      "watcher_state" : "stopped",
      "watch_count" : 0,
      "execution_thread_pool" : {
        "queue_size" : 0,
        "max_size" : 10
      }
    }
  ]

Maybe my issue has something to do with "watcher_state" : "stopped",

5

Found a solution for this, if anyone else is interested:

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.