Elasticsearch Watcher for multiple hosts

[Nishant_Suman]

Hi,

I am new to the elastic stack and trying to set up some basic alerts based on server metrics. The stack that I have configured will be used as a centralized logging system. Right now I am setting up watcher for each server, so if there are 4 servers that needed to be monitored(let's say disk usage), I am setting up 4 watchers(4 disk usage watcher for 4 different servers) for each server. Is there any other way where I can set up a generic watcher (1 watcher for disk usage for all servers)? So while sending alerts if two servers have disk usage greater than the threshold, I should receive 2 different alerts.

My current configuration:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "metricbeat-*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now"
                    }
                  }
                },
                {
                  "match": {
                    "beat.hostname": "CALM-POD"
                  }
                }
              ]
            }
          },
          "aggs": {
            "DiskUsed": {
              "max": {
                "field": "system.fsstat.total_size.used"
              }
            },
            "DiskTotal": {
              "max": {
                "field": "system.fsstat.total_size.total"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "if (ctx.payload.aggregations.DiskUsed.value / ctx.payload.aggregations.DiskTotal.value > params.threshold) { return true; } return false;",
      "lang": "painless",
      "params": {
        "threshold": 0.8
      }
    }
  },
  "actions": {
    "notify-slack": {
      "throttle_period_in_millis": 300000,
      "slack": {
        "message": {
          "to": [
            "#elk-alerts-test"
          ],
          "text": "Test watcher of Disk usage for CALM-POD has exceeded the threshold of 85%."
        }
      }
    }
  },
  "transform": {
    "script": {
      "source": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.CPUAggs.value; return result;",
      "lang": "painless",
      "params": {
        "threshold": 0.8
      }
    }
  }
}

[spinscale (Alexander Reelsen)]

Hey,

you can nest aggregations. So what you could do instead would be to have a terms aggregation on the beat.hostname field and then nest your DiskUsed and DiskTotal aggregagations within that terms aggregation.

The response then would then include aggregation buckets per host, containing that data.

You would also have to change your condition, as only a single bucket matching the criteria should trigger the alert.

Lastly I also suppose you would need a so called transform in your action, as you only want to notify about the hosts that match your condition criteria.]

Hope this helps!

Just keep in mind, that if you want to have different alerts per hosts or different trigger conditions, than it might make sense to have an alert per host (or another grouping scheme using the above strategy).

--Alex

[Nishant_Suman]

Got it. Can you provide a sample or any link on how to do it?

[spinscale (Alexander Reelsen)]

Check out our alerting examples in the examples repo as a starter.

[Nishant_Suman]

Thanks.