Alert when watcher is in error state - .watcher_history

Elastic Stack Elasticsearch
1

Hi, we have a number of watchers that send notification to rest_api (Service.now) to create an incident.
I would like to create a generic alert that if a watcher cannot send notification or is in Error state (Execution Failing) would send an email out.

The idea is to create a watcher digesting .watcher_history indices but I not sure what a field within that record to use query.

Any help would be appreciated.

2

Hey,

not everything in the watcher history is searchable, so you need to take a look at the mapping to be sure you can actually search and filter on the values you are trying to extract data from.

Do you have a sample query we can work with and see if you can search on the fields you want?

--Alex

3

Hi Alex, this is a payload from .watcher-history with the field status.actions.my_webhook.last_execution.successful that if false to trigger an action. Unfortunately currently this field is not indexed. We tried the fields rediscovery and although brought it up from ~130 to 170 fields the one I am interested has not appeared there yet. I have been thinking to create Explicit Mapping but I am still newbie to this and currently researching this option.

{
"watch_id": "errors_capabilities_HRAAAA",
"node": "AAAAvQsmSSuieKzuOlBBBB",
"state": "execution_not_needed",
"status": {
"state": {
"active": true,
"timestamp": "2018-06-19T10:59:15.845Z"
},
"last_checked": "2018-06-22T09:44:42.139Z",
"last_met_condition": "2018-06-19T11:01:16.589Z",
"actions": {
"my_webhook": {
"ack": {
"timestamp": "2018-06-19T10:59:15.845Z",
"state": "awaits_successful_execution"
},
"last_execution": {
"timestamp": "2018-06-19T11:01:16.589Z",
"successful": false,
"reason": ""
}
},
"notify-slack": {
"ack": {
"timestamp": "2018-06-19T11:02:18.143Z",
"state": "awaits_successful_execution"
},
"last_execution": {
"timestamp": "2018-06-19T11:01:16.589Z",
"successful": true
},
"last_successful_execution": {
"timestamp": "2018-06-19T11:01:16.589Z",
"successful": true
}
}
},
"execution_state": "execution_not_needed",
"version": -1
}

4

I am including kinda related topic I raised I cannot index "status.actions.my_webhook.last_execution.successful" in .watcher-history-* about unable to index a field.

To be honest all I want is programmatic method to alert if a watcher fails.

5

Hey,

if you search for the last 10 watch history entries for this watch, then you do not need to search for the last execution flag. You can just use painless scripting in the condition to check the value of the last 10 entries that are being returned as part of the search.

Hope that makes sense.

--Alex

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.