# Alert when watcher is in error state - .watcher\_history

**URL:** <https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [June 18, 2018, 4:59pm UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348 "2018-06-18T16:59:37Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Piotr\_Pawlak](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@Piotr\_Pawlak](https://discuss.elastic.co/u/Piotr_Pawlak)\
**Post date:** [June 18, 2018, 4:59pm UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348/1 "2018-06-18T16:59:37Z")

</div>

Hi, we have a number of watchers that send notification to rest\_api (Service.now) to create an incident.  
I would like to create a generic alert that if a watcher cannot send notification or is in Error state (Execution Failing) would send an email out.

The idea is to create a watcher digesting .watcher\_history indices but I not sure what a field within that record to use query.

Any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 22, 2018, 2:06pm UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348/2 "2018-06-22T14:06:00Z")

</div>

Hey,

not everything in the watcher history is searchable, so you need to take a look at the mapping to be sure you can actually search and filter on the values you are trying to extract data from.

Do you have a sample query we can work with and see if you can search on the fields you want?

--Alex

---

<div class="post-metadata">

**Author:** ![Piotr\_Pawlak](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@Piotr\_Pawlak](https://discuss.elastic.co/u/Piotr_Pawlak)\
**Post date:** [June 22, 2018, 3:05pm UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348/3 "2018-06-22T15:05:24Z")

</div>

Hi Alex, this is a payload from .watcher-history with the field `status.actions.my_webhook.last_execution.successful` that if `false` to trigger an action. Unfortunately currently this field is not indexed. We tried the fields rediscovery and although brought it up from ~130 to 170 fields the one I am interested has not appeared there yet. I have been thinking to create Explicit Mapping but I am still newbie to this and currently researching this option.

{  
"watch\_id": "errors\_capabilities\_HRAAAA",  
"node": "AAAAvQsmSSuieKzuOlBBBB",  
"state": "execution\_not\_needed",  
"status": {  
"state": {  
"active": true,  
"timestamp": "2018-06-19T10:59:15.845Z"  
},  
"last\_checked": "2018-06-22T09:44:42.139Z",  
"last\_met\_condition": "2018-06-19T11:01:16.589Z",  
"actions": {  
"my\_webhook": {  
"ack": {  
"timestamp": "2018-06-19T10:59:15.845Z",  
"state": "awaits\_successful\_execution"  
},  
"last\_execution": {  
"timestamp": "2018-06-19T11:01:16.589Z",  
"successful": false,  
"reason": ""  
}  
},  
"notify-slack": {  
"ack": {  
"timestamp": "2018-06-19T11:02:18.143Z",  
"state": "awaits\_successful\_execution"  
},  
"last\_execution": {  
"timestamp": "2018-06-19T11:01:16.589Z",  
"successful": true  
},  
"last\_successful\_execution": {  
"timestamp": "2018-06-19T11:01:16.589Z",  
"successful": true  
}  
}  
},  
"execution\_state": "execution\_not\_needed",  
"version": -1  
}

---

<div class="post-metadata">

**Author:** ![Piotr\_Pawlak](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@Piotr\_Pawlak](https://discuss.elastic.co/u/Piotr_Pawlak)\
**Post date:** [June 22, 2018, 3:25pm UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348/4 "2018-06-22T15:25:52Z")

</div>

I am including kinda related topic I raised [I cannot index "status.actions.my\_webhook.last\_execution.successful" in .watcher-history-\*](https://discuss.elastic.co/t/i-cannot-index-status-actions-my-webhook-last-execution-successful-in-watcher-history/137038/2?u=piotr_pawlak) about unable to index a field.

To be honest all I want is programmatic method to alert if a watcher fails.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 25, 2018, 9:06am UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348/5 "2018-06-25T09:06:57Z")

</div>

Hey,

if you search for the last 10 watch history entries for this watch, then you do not need to **search** for the last execution flag. You can just use painless scripting in the condition to check the value of the last 10 entries that are being returned as part of the search.

Hope that makes sense.

--Alex

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 23, 2018, 9:07am UTC](https://discuss.elastic.co/t/alert-when-watcher-is-in-error-state-watcher-history/136348/6 "2018-07-23T09:07:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.