# ELK Watcher Alarms

**URL:** <https://discuss.elastic.co/t/elk-watcher-alarms/213566>\
**Category:** Elasticsearch\
**Created:** [January 2, 2020, 11:43am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566 "2020-01-02T11:43:37Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rj23495](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@rj23495](https://discuss.elastic.co/u/rj23495)\
**Post date:** [January 2, 2020, 11:43am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/1 "2020-01-02T11:43:37Z")

</div>

Hey, I am trying to create alarms in elk watcher with the following parameters like 5 datapoints in 20 min for value \> 10 but i cannot see anything apart from average median, etc. How can we achieve this type pf alarm config in elk watcher.

---

<div class="post-metadata">

**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\
**Post date:** [January 2, 2020, 9:31pm UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/2 "2020-01-02T21:31:12Z")

</div>

You will need to create an "advanced watch" via Kibana , or use the [PUT watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-put-watch.html).

If your 5 data points can be achieved with a single \_search query and/or aggregation, you should look at the [search input](https://www.elastic.co/guide/en/elasticsearch/reference/current/input-search.html) with a [compare condition](https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-compare.html). There is a full example [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/watching-meetup-data.html)

If you need to pull your 5 data points from multiple queries or aggregations you may need to use the [chained input](https://www.elastic.co/guide/en/elasticsearch/reference/current/input-chain.html) and [script condition](https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-script.html) to handle the complexity.

---

<div class="post-metadata">

**Author:** ![rj23495](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@rj23495](https://discuss.elastic.co/u/rj23495)\
**Post date:** [January 3, 2020, 6:56am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/3 "2020-01-03T06:56:41Z")

</div>

Thanks @jakelandis i will go through the links and try them out and will let you know in case of any issues.

---

<div class="post-metadata">

**Author:** ![rj23495](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@rj23495](https://discuss.elastic.co/u/rj23495)\
**Post date:** [January 6, 2020, 10:37am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/4 "2020-01-06T10:37:37Z")

</div>

I have formulated this query for an index, now i can get the count using ctx.payload.hits right?

```
{
"query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-3h"
                    }
                  }
                },
                {
                  "match": {
                    "metricname": "Api_success_latency" 
                  }
                },
                {
                  "range": {
                    "metricvalue": {
                      "gte": 17
                    }
                  }
                }
              ]
            }
          }
}
```

---

<div class="post-metadata">

**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\
**Post date:** [January 6, 2020, 3:10pm UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/5 "2020-01-06T15:10:33Z")

</div>

> now i can get the count using ctx.payload.hits right?

Correct, if all you need is the hits, then you can put that in a [compare condition](https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-compare.html) .

---

<div class="post-metadata">

**Author:** ![rj23495](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@rj23495](https://discuss.elastic.co/u/rj23495)\
**Post date:** [January 8, 2020, 10:27am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/6 "2020-01-08T10:27:55Z")

</div>

In this context what will be the difference between hits and count?

---

<div class="post-metadata">

**Author:** ![rj23495](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@rj23495](https://discuss.elastic.co/u/rj23495)\
**Post date:** [January 14, 2020, 9:45am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/7 "2020-01-14T09:45:41Z")

</div>

This is an alarm i created but it is showing condition failed skipping runtime -  
{  
"trigger": {  
"schedule": {  
"interval": "1m"  
}  
},  
"input": {  
"search": {  
"request": {  
"search\_type": "query\_then\_fetch",  
"indices": [  
"prod-queueworker-service-\*"  
],  
"types": ,  
"body": {  
"size": 0,  
"query": {  
"bool": {  
"filter": {  
"range": {  
"@timestamp": {  
"gte": "{{ctx.trigger.scheduled\_time}}||-5m",  
"lte": "{{ctx.trigger.scheduled\_time}}",  
"format": "strict\_date\_optional\_time||epoch\_millis"  
}  
}  
}  
}  
},  
"aggs": {  
"metricAgg": {  
"avg": {  
"field": "metric\_lqs-available-messages\_QueueName\_vae-plan-dead-letter-queue"  
}  
}  
}  
}  
}  
}  
},  
"condition": {  
"always": {}  
},  
"actions": {  
"pagerduty": {  
"throttle\_period\_in\_millis": 1800000,  
"condition": {  
"script": {  
"source": "if (ctx.payload.aggregations.metricAgg.value \> params.threshold) { return true; } return false;",  
"lang": "painless",  
"params": {  
"threshold": 1  
}  
}  
},  
"webhook": {  
"scheme": "https",  
"host": "[events.pagerduty.com](http://events.pagerduty.com)",  
"port": 443,  
"method": "post",  
"path": "/generic/2010-04-15/create\_event.json",  
"params": {},  
"headers": {  
"Content-type": "application/json"  
},  
"body": "{"service\_key": "abcd","incident\_key": "vaedlq","event\_type": "trigger","description": "Breaching threshold for LAZADA for VAE general DLQ"}"  
}  
},  
"pagerduty-resolve": {  
"condition": {  
"script": {  
"source": "if (ctx.payload.aggregations.metricAgg.value \< params.threshold) { return true; } return false;",  
"lang": "painless",  
"params": {  
"threshold": 1  
}  
}  
},  
"webhook": {  
"scheme": "https",  
"host": "[events.pagerduty.com](http://events.pagerduty.com)",  
"port": 443,  
"method": "post",  
"path": "/generic/2010-04-15/create\_event.json",  
"params": {},  
"headers": {  
"Content-type": "application/json"  
},  
"body": "{"service\_key": "abcd","incident\_key": "vaedlq","event\_type": "resolve","description": "Issue resolved"}"  
}  
}  
}  
}

output - ---------------------------------------------------------------------

```
{
  "watch_id": "lqs-available-messages_QueueName_taxy-general-dead-letter-queue",
  "node": "Ud7KVIhsTfe44UC0q4540w",
  "state": "executed",
  "user": "elastic",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2020-01-13T10:38:46.225Z"
    },
    "last_checked": "2020-01-14T09:43:36.205Z",
    "last_met_condition": "2020-01-14T09:43:36.205Z",
    "actions": {
      "pagerduty-resolve": {
        "ack": {
          "timestamp": "2020-01-13T10:38:46.225Z",
          "state": "awaits_successful_execution"
        }
      },
      "pagerduty": {
        "ack": {
          "timestamp": "2020-01-13T10:39:36.137Z",
          "state": "ackable"
        },
        "last_execution": {
          "timestamp": "2020-01-13T10:39:36.137Z",
          "successful": true
        },
        "last_successful_execution": {
          "timestamp": "2020-01-13T10:39:36.137Z",
          "successful": true
        },
        "last_throttle": {
          "timestamp": "2020-01-13T11:08:36.303Z",
          "reason": "throttling interval is set to [30m] but time elapsed since last execution is [29m]"
        }
      }
    },
    "execution_state": "executed",
    "version": -1
  },
  "trigger_event": {
    "type": "schedule",
    "triggered_time": "2020-01-14T09:43:36.205Z",
    "schedule": {
      "scheduled_time": "2020-01-14T09:43:35.980Z"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "prod-queueworker-service-*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                    "lte": "{{ctx.trigger.scheduled_time}}",
                    "format": "strict_date_optional_time||epoch_millis"
                  }
                }
              }
            }
          },
          "aggs": {
            "metricAgg": {
              "avg": {
                "field": "metric_lqs-available-messages_QueueName_taxy-general-dead-letter-queue"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "always": {}
  },
  "metadata": {
    "name": "lqs-available-messages_QueueName_taxy-general-dead-letter-queue",
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2020-01-14T09:43:36.205Z",
    "execution_duration": 15,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 10,
          "failed": 0,
          "successful": 10,
          "skipped": 0
        },
        "hits": {
          "hits": [],
          "total": 225188,
          "max_score": 0
        },
        "took": 11,
        "timed_out": false,
        "aggregations": {
          "metricAgg": {
            "value": null
          }
        }
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "prod-queueworker-service-*"
          ],
          "types": [],
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": {
                  "range": {
                    "@timestamp": {
                      "gte": "2020-01-14T09:43:35.980Z||-5m",
                      "lte": "2020-01-14T09:43:35.980Z",
                      "format": "strict_date_optional_time||epoch_millis"
                    }
                  }
                }
              }
            },
            "aggs": {
              "metricAgg": {
                "avg": {
                  "field": "metric_lqs-available-messages_QueueName_taxy-general-dead-letter-queue"
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "always",
      "status": "success",
      "met": true
    },
    "actions": [
      {
        "id": "pagerduty-resolve",
        "type": "webhook",
        "status": "condition_failed",
        "reason": "condition failed. skipping: runtime error"
      },
      {
        "id": "pagerduty",
        "type": "webhook",
        "status": "condition_failed",
        "reason": "condition failed. skipping: runtime error"
      }
    ]
  },
  "messages": []
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 11, 2020, 9:46am UTC](https://discuss.elastic.co/t/elk-watcher-alarms/213566/8 "2020-02-11T09:46:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.