How can I successfully debug a watcher? My watchers don't seem to do anything

[ujjain]

My watchers don´t seem to do anything. Nothing appears in the _alerts index and no e-mail gets sent. I'm not sure what effective methods of debugging are either.

I'm running ElasticSearch 2.4.

{
  "_index": "alert_watchers",
  "_type": "watcher",
  "_id": "Test Alerts 2",
  "_score": 1,
  "_source": {
    "name": "Test Alerts 2",
    "disabled": false,
    "status": "OK",
    "loglevel": "ERROR",
    "dashboard": "https://kibana",
    "tags": "demo",
    "trigger": {
      "schedule": {
        "later": "every 5 minutes",
        "plan": "DAILY",
        "sun": {
          "from": "07:00",
          "to": "19:00"
        },
        "mon": {
          "from": "07:00",
          "to": "19:00"
        },
        "tue": {
          "from": "07:00",
          "to": "19:00"
        },
        "wed": {
          "from": "07:00",
          "to": "19:00"
        },
        "thu": {
          "from": "07:00",
          "to": "19:00"
        },
        "fri": {
          "from": "07:00",
          "to": "19:00"
        },
        "sat": {
          "from": "07:00",
          "to": "19:00"
        }
      }
    },
    "input": {
      "search": {
        "request": {
          "index": "<logstash-{now/d}>",
          "body": {
            "highlight": {
              "pre_tags": [
                "@kibana-highlighted-field@"
              ],
              "post_tags": [
                "@/kibana-highlighted-field@"
              ],
              "fields": {
                "*": {}
              },
              "require_field_match": false,
              "fragment_size": 2147483647
            },
            "query": {
              "filtered": {
                "query": {
                  "query_string": {
                    "query": "response:404",
                    "analyze_wildcard": true
                  }
                },
                "filter": {
                  "range": {
                    "@timestamp": {
                      "from": "now-5m"
                    }
                  }
                }
              }
            },
            "size": 0,
            "aggs": {
              "2": {
                "date_histogram": {
                  "field": "@timestamp",
                  "interval": "10m",
                  "time_zone": "Europe/Berlin",
                  "min_doc_count": 1,
                  "extended_bounds": {
                    "min": 1519207582152,
                    "max": 1519293982152
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 1"
      }
    },
    "record_execution": true,
    "ignore_condition": true,
    "action_modes": {
      "my-action": "force_simulate"
    },
    "transform": {},
    "actions": {
      "action_group": {
        "throttle_period": "1m",
        "console": {
          "message": "error in demo2"
        },
        "email": {
          "to": "me@me.com",
          "subject": "error for demo2",
          "body": "Number of 404 errors last 5 minutes is  {{ payload.hits.total }}. Threshold for error is 1 error last 5 minutes."
        },
        "chat": {
          "hook": "",
          "message": ""
        },
        "webhook": {
          "host": "",
          "port": "",
          "method": "",
          "path": "",
          "headers": "",
          "body": ""
        }
      }
    },
    "@timestamp": "2018-03-02T09:43:31.289Z"
  }
}

[LeeDr (Lee Drengenberg)]

I'm not sure how much has changed since 2.4 version, but in current 6.x versions I test a watch with email like this;

1. Create the watch
2. periodically get the watch from Elasticsearch from .watches index (may not be the same in 2.4)
3. if the watch does not show successful execution, then I search .watcher-history* with 'watch_id:' myWatchId (also may not be the same in 2.4)

For example, if the watch fails to execute the email action, the doc in the .watches index doesn't usually have the details of why it failed. But the .watcher-history* entry does.

Also note that you can add index patterns to Kibana for these indices and use it to look at the results.

[spinscale (Alexander Reelsen)]

This post here might help, even though it is written for Elasticsearch 5.x and thus some endpoints differ... https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches

First thing is to always use the execute watch API and check the response output... the above blogpost explains a bit how to read the output.