# Alert When - system\_fails\_to\_provide\_data - dynamic group of sending hosts

**URL:** <https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094>\
**Category:** Logs\
**Created:** [February 22, 2021, 4:51pm UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094 "2021-02-22T16:51:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bbek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bbek/32/44087_2.png) [@bbek](https://discuss.elastic.co/u/bbek)\
**Post date:** [February 22, 2021, 4:51pm UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/1 "2021-02-22T16:51:31Z")

</div>

I have a use case to alert when a host fails to send logs. There is a watcher configured here, which is similar to what I'm trying to achieve:

> **[elastic/examples](https://github.com/elastic/examples/tree/master/Alerting/Sample%20Watches/system_fails_to_provide_data)**
>
> master/Alerting/Sample%20Watches/system\_fails\_to\_provide\_data

I like the logic, aggregate hosts on last 24 hours, then check for last 5 minutes.

However when trying to modify for filebeat-\*, I am getting an error.  
Is using watcher the best form of alerting, or is there a simpler/more elegant approach?

 ![dynamic_sending_hosts_outage](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e88c833612205afa7bb3624bb0a03fd5e5ec3688.png)

---

<div class="post-metadata">

**Author:** ![Andrew\_Cholakian1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_cholakian1/32/3612_2.png) [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)\
**Post date:** [February 24, 2021, 8:39pm UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/2 "2021-02-24T20:39:23Z")

</div>

I've moved this to the logs category, it was posted to Uptime which covers a different area.

---

<div class="post-metadata">

**Author:** ![bbek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bbek/32/44087_2.png) [@bbek](https://discuss.elastic.co/u/bbek)\
**Post date:** [March 1, 2021, 10:48pm UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/3 "2021-03-01T22:48:45Z")

</div>

Thanks sir. If I can be any more clear in the ask, I'll be happy to elaborate.

Seeking to obtain notice when 1 in N hosts ceases to send logs.

Where N may be 2k-3k.

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [March 8, 2021, 3:25pm UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/4 "2021-03-08T15:25:53Z")

</div>

Hi @bbek,

there's now an alerting functionality built into Kibana, that's intended to be simpler to work with than watcher. You could try to implement this using the "Log threshold" alert type. The limitation is that it provides less control over the specific semantics of the condition.

Regarding the error message you're seeing, it seems that the `condition` script from the example repo hasn't been added to the cluster.

---

<div class="post-metadata">

**Author:** ![bbek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bbek/32/44087_2.png) [@bbek](https://discuss.elastic.co/u/bbek)\
**Post date:** [March 8, 2021, 9:04pm UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/5 "2021-03-08T21:04:28Z")

</div>

Understood now! Appreciate the clarity.

I'm making progress toward fetching the "observer.names" (syslog sending hosts) - and appears script is demonstrating results.

I need some assistance in formatting and sharpening the output.

How do we extract the '`observer.name`' - so that I customize an alert upon "match" of this (alerter approach is one way)

OR

Execute an email from Watcher , inclusive of the "flatlined" sending `'observer.name'` when conditions match?

I realize the `{{ctx.payload.hits}}` is not the field I need, but it's the location I'd like the the `'observer.name'` to be. Is it `'_source.observer.name'` ?

//  
`"text": "Systems not responding in the last {{ctx.metadata.last_period}} minutes:{{#ctx.payload._value}}{{.}}:{{/ctx.payload._value}} {{ctx.payload.hits}}`  
//

The watcher json:

```
{
  "metadata": {
    "window_period": "24h",
    "last_period":"5m"
  },
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": "filebeat-*",
        "body": {
          "query": {
            "range": {
              "@timestamp": {
                "gte": "now-{{ctx.metadata.window_period}}"
              }
            }
          },
          "aggs": {
            "periods": {
              "filters": {
                "filters": {
                  "history": {
                    "range": {
                      "@timestamp": {
                        "gte": "now-{{ctx.metadata.window_period}}"
                      }
                    }
                  },
                  "last_period": {
                    "range": {
                      "@timestamp": {
                        "gte": "now-{{ctx.metadata.last_period}}"
                      }
                    }
                  }
                }
              },
              "aggs": {
                "hosts": {
                  "terms": {
                    "field": "observer.name",
                    "size": 10000
                  }
                }
              }
            }
          },
          "size": 0
        }
      }
    }
  },
  "condition": {
    "script": "return true"
  },
  "throttle_period": "5m",
  "actions": {
    "log": {
      "transform": {
      "script": "return true"
    },
      "logging": {
        "text": "Systems not responding in the last {{ctx.metadata.last_period}} minutes:{{#ctx.payload._value}}{{.}}:{{/ctx.payload._value}} {{ctx.payload.hits}}"
      }
    }
  }
}

```

Simulated Results:

```
> {
> "watch_id": "_inlined_",
> "node": "_KlItRP8Qp27WSYwbMYzsw",
> "state": "executed",
> "user": "elastic",
> "status": {
> "state": {
> "active": true,
> "timestamp": "2021-03-08T20:49:58.502Z"
> },
> "last_checked": "2021-03-08T20:49:58.503Z",
> "last_met_condition": "2021-03-08T20:49:58.503Z",
> "actions": {
> "log": {
> "ack": {
> "timestamp": "2021-03-08T20:49:58.503Z",
> "state": "ackable"
> },
> "last_execution": {
> "timestamp": "2021-03-08T20:49:58.503Z",
> "successful": true
> },
> "last_successful_execution": {
> "timestamp": "2021-03-08T20:49:58.503Z",
> "successful": true
> }
> }
> },
> "execution_state": "executed",
> "version": -1
> },
> "trigger_event": {
> "type": "manual",
> "triggered_time": "2021-03-08T20:49:58.503Z",
> "manual": {
> "schedule": {
> "scheduled_time": "2021-03-08T20:49:58.503Z"
> }
> }
> },
> "input": {
> "search": {
> "request": {
> "search_type": "query_then_fetch",
> "indices": [
> "filebeat-*"
> ],
> "rest_total_hits_as_int": true,
> "body": {
> "query": {
> "range": {
> "@timestamp": {
> "gte": "now-{{ctx.metadata.window_period}}"
> }
> }
> },
> "aggs": {
> "periods": {
> "filters": {
> "filters": {
> "history": {
> "range": {
> "@timestamp": {
> "gte": "now-{{ctx.metadata.window_period}}"
> }
> }
> },
> "last_period": {
> "range": {
> "@timestamp": {
> "gte": "now-{{ctx.metadata.last_period}}"
> }
> }
> }
> }
> },
> "aggs": {
> "hosts": {
> "terms": {
> "field": "observer.name",
> "size": 10000
> }
> }
> }
> }
> },
> "size": 0
> }
> }
> }
> },
> "condition": {
> "script": {
> "source": "return true",
> "lang": "painless"
> }
> },
> "metadata": {
> "last_period": "5m",
> "window_period": "24h",
> "name": "testtest",
> "xpack": {
> "type": "json"
> }
> },
> "result": {
> "execution_time": "2021-03-08T20:49:58.503Z",
> "execution_duration": 472,
> "input": {
> "type": "search",
> "status": "success",
> "payload": {
> "_shards": {
> "total": 53,
> "failed": 0,
> "successful": 53,
> "skipped": 49
> },
> "hits": {
> "hits": [],
> "total": 10000,
> "max_score": null
> },
> "took": 470,
> "timed_out": false,
> "aggregations": {
> "periods": {
> "buckets": {
> "last_period": {
> "doc_count": 42242,
> "hosts": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "doc_count": 3619,
> "key": "FortiGate-60F"
> },
> {
> "doc_count": 3524,
> "key": "NC-Firewall"
> },
> {
> "doc_count": 982,
> "key": "TexasFG"
> },
> {
> "doc_count": 4,
> "key": "ok-int-wall2600"
> }
> ]
> }
> },
> "history": {
> "doc_count": 7415336,
> "hosts": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "doc_count": 519864,
> "key": "FortiGate-60F"
> },
> {
> "doc_count": 400606,
> "key": "NC-Firewall"
> },
> {
> "doc_count": 148454,
> "key": "TexasFG"
> },
> {
> "doc_count": 1176,
> "key": "ok-int-wall2600"
> }
> ]
> }
> }
> }
> }
> }
> },
> "search": {
> "request": {
> "search_type": "query_then_fetch",
> "indices": [
> "filebeat-*"
> ],
> "rest_total_hits_as_int": true,
> "body": {
> "query": {
> "range": {
> "@timestamp": {
> "gte": "now-24h"
> }
> }
> },
> "aggs": {
> "periods": {
> "filters": {
> "filters": {
> "history": {
> "range": {
> "@timestamp": {
> "gte": "now-24h"
> }
> }
> },
> "last_period": {
> "range": {
> "@timestamp": {
> "gte": "now-5m"
> }
> }
> }
> }
> },
> "aggs": {
> "hosts": {
> "terms": {
> "field": "observer.name",
> "size": 10000
> }
> }
> }
> }
> },
> "size": 0
> }
> }
> }
> },
> "condition": {
> "type": "script",
> "status": "success",
> "met": true
> },
> "actions": [
> {
> "id": "log",
> "type": "logging",
> "status": "simulated",
> "transform": {
> "type": "script",
> "status": "success",
> "payload": {
> "_value": true
> }
> },
> "logging": {
> "logged_text": "Systems not responding in the last 5m minutes:true: "
> }
> }
> ]
> },
> "messages": []
> }
```

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [March 9, 2021, 9:08am UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/6 "2021-03-09T09:08:48Z")

</div>

The top-level `condition` and action-level `transform` seem to play the following roles in this watch:

- The `condition` compares the current period's host list length with the reference period's host list length and causes the action to trigger when the current host list's length is smaller. With your change to the `"hosts"` `terms agg` you defined that a distinct host is identified via its `observer.name` field. Since the example script only refers to the aggregation names it shouldn't require any change to work.
- The action-level `transform` extracts the list of "missing" host identifiers from the response to make it available in the message. Since it just refers to the `key` of each `terms agg` bucket it shouldn't require any change and return a list of `observer.name` values.

Given this I wonder why it wouldn't work with the scripts included in the example. 🤔 Can you try and tell us the (simulated) results you got?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 2, 2021, 9:08am UTC](https://discuss.elastic.co/t/alert-when-system-fails-to-provide-data-dynamic-group-of-sending-hosts/265094/7 "2021-04-02T09:08:54Z")

</div>

This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.