# Create a simple frequency based alert using Watcher
**URL:** https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [August 8, 2018, 7:56pm UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586 "2018-08-08T19:56:13Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@rouble](https://discuss.elastic.co/u/rouble)
#### Post date: [August 8, 2018, 7:56pm UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/1 "2018-08-08T19:56:13Z")
Gurus,
I am looking to create a frequency based watch. Very simply, if the query returns any results at all in the last N seconds, send an alert. I have written something that works, but since I am new to writing watches, I feel like what I have may be overcomplicated. In this example, I have N as 30 seconds, so I have the schedule interval set to 30s, and then I have a range filter on @timestamp: "gte": "now-30s".
My watch is pasted below. Two questions, is there a more efficient way to do this? Secondly, is it possible for me to miss query matches.
```
{
"trigger": {
"schedule": {
"interval": "30s"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash-*"
],
"types": [],
"body": {
"size": 0,
"query": {
"range": {
"@timestamp": {
"gte": "now-30s"
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 1
}
}
},
"actions": {
"my-logging-action": {
"logging": {
"level": "info",
"text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 1."
}
}
}
}
```
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [August 9, 2018, 6:48am UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/2 "2018-08-09T06:48:29Z")
Hey,
the watch itself looks fine. The discussion around missing query matches however is a completely different one. I have two things to add to that
1. Elasticsearch has something called a [refresh interval](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings), which defines how often data is made available for search (by default every second). This means, that your data might only be available from the last 29 seconds, but when doing the next query it is no more in that time window
2. The above however is only a minor problem. The IMO bigger issue is the fact, that the timestamp is usually the timestamp when the event has been created within the application. What is not taken account for, is the fact that this event needs to travel to Elasticsearch. Maybe you are sending data directly from beats to ES, but maybe you are sending it to a broker first, where it sits a few seconds and then it gets indexed. Also your ingestion could have a bigger delay due to a DDoS attack or network outage. This will add a bigger delay than the one second refresh above.
The question then is, are you good with ignoring those things, or do you want to query bigger time windows with the likelihood of duplicating alerts or add some more fancy mechanism for alerting.
Hope this helps!
--Alex
---
### Author:  [@rouble](https://discuss.elastic.co/u/rouble)
#### Post date: [August 15, 2018, 6:08pm UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/3 "2018-08-15T18:08:25Z")
Thank you. That helps. To summarize what you said, if the watch runs every 30 seconds, but queries data for 30+N seconds worth of data, we should not lose any hits, but we may, occasionally, get some duplicates.
This kind of watch is very commonly written in alerting systems, so that we don't inundate the receiver of the alarm or the notification. It is usually called notification interval or alarm interval.
Here is the definition notification\_interval from another, now ancient, notification framework called nagios:
**notification\_interval** : This directive is used to define the number of "time units" to wait before re-notifying a contact that this service is _still_ down or unreachable.
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [August 17, 2018, 8:44am UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/4 "2018-08-17T08:44:30Z")
I think you may be interested in the acknowledgement/throttling capabilities of watcher in that context. Please see
[https://www.elastic.co/guide/en/elastic-stack-overview/6.3/actions.html#actions-ack-throttle](https://www.elastic.co/guide/en/elastic-stack-overview/6.3/actions.html#actions-ack-throttle)
[https://www.elastic.co/guide/en/elastic-stack-overview/6.3/how-watcher-works.html#watch-acknowledgment-throttling](https://www.elastic.co/guide/en/elastic-stack-overview/6.3/how-watcher-works.html#watch-acknowledgment-throttling)
---
### Author:  [@rouble](https://discuss.elastic.co/u/rouble)
#### Post date: [August 17, 2018, 12:57pm UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/5 "2018-08-17T12:57:43Z")
Nice! **throttle\_period** looks very interesting. Any thoughts if there is a way to better rewrite my watch using throttle\_period. Here is essentially what I need to do:
_If there are any **new** hits to my query in the last **N seconds** do the appropriate action._
We are trying to use watcher to alert via pagerduty when high severity logs come through.
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [August 20, 2018, 7:28am UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/6 "2018-08-20T07:28:53Z")
watches are running stateless so the definition of `new` is a tricky one. If you need state, you could always store the result count of a query in its own document using the `index` action, and compare that count at the next run of a watch, when running the same query with the same filters.
Another alternative could be, that if those documents are super rare, you store information, if you already processed these documents, but that is not feasible for higher volumes.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 17, 2018, 7:28am UTC](https://discuss.elastic.co/t/create-a-simple-frequency-based-alert-using-watcher/143586/7 "2018-09-17T07:28:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.