# Watching watcher
**URL:** https://discuss.elastic.co/t/watching-watcher/1002
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [May 20, 2015, 5:00pm UTC](https://discuss.elastic.co/t/watching-watcher/1002 "2015-05-20T17:00:04Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@jbaranick](https://discuss.elastic.co/u/jbaranick)
#### Post date: [May 20, 2015, 5:00pm UTC](https://discuss.elastic.co/t/watching-watcher/1002/1 "2015-05-20T17:00:04Z")
It would be nice to be able to configure watches to alert when the queries fail to return within a configured timeout. The default timeout could be the schedule duration. This protects against long running queries causing watches to not fire.
---
### Author:  [@skearns](https://discuss.elastic.co/u/skearns)
#### Post date: [May 21, 2015, 9:16am UTC](https://discuss.elastic.co/t/watching-watcher/1002/2 "2015-05-21T09:16:43Z")
This is a good question - I would love to learn more about your goal here. I imagine a few reasons:
- Get notified when any configured watches exceed their configured timeouts
- Ensure consistent watch execution times for short-interval watches
Are there other goals you had in mind?
Today, you can specify a timeout in the search input request body, and we do record the search execution information (e.g. `execution_result.input.search.payload.took`, and `timed_out`) in the watch history. These fields aren't indexed today, but it's something we could consider adding, so you could create a watch that looks at the watch history for timed\_out ES queries.
---
### Author:  [@jbaranick](https://discuss.elastic.co/u/jbaranick)
#### Post date: [May 21, 2015, 1:31pm UTC](https://discuss.elastic.co/t/watching-watcher/1002/3 "2015-05-21T13:31:04Z")
If we are relying on watches to alert us when there is a production issue, then the watches are a critical piece of the infrastructure. As such, if watches stop running (or start timing out) it is a production live site which needs to be immediately addressed. This means we need to be alerted about watches which timeout, fail, or fail to run. Ideally, this notification would be resistant to elastricsearch cluster issues (red, lots of GC-ing, etc.).
---
### Author:  [@skearns](https://discuss.elastic.co/u/skearns)
#### Post date: [June 1, 2015, 9:00pm UTC](https://discuss.elastic.co/t/watching-watcher/1002/4 "2015-06-01T21:00:24Z")
Joel,
Coming back around to this. I now see what you're after, and it makes a lot of sense.
Many of our customers use Marvel for monitoring Elasticsearch - it records metrics and telemetry from Elasticsearch over time. For larger clusters, we recommend storing the Marvel data in a separate monitoring cluster. In much the same way, you can run Watcher on a monitoring cluster and simply query your production cluster using the HTTP input:
[https://www.elastic.co/guide/en/watcher/current/anatomy-input.html#anatomy-input-http](https://www.elastic.co/guide/en/watcher/current/anatomy-input.html#anatomy-input-http)
We expected that monitoring Elasticsearch itself would be a common use-case, so we have provided a few examples of watches based on Marvel data:
[https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html](https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html)
Hope that helps!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:49pm UTC](https://discuss.elastic.co/t/watching-watcher/1002/5 "2017-07-06T13:49:45Z")