Extract a tag field into watcher / Watcher to Alert if a Heartbeat Host is Down

alerting

(Mohamed Salih) #1

I am trying to create a watcher, and I am using Heartbeat to check if any hosts are down.

I have add a tag field to the IPs that I will be monitoring, this helps identify them easier. However, the tag field is not in the payload.
I would like the message to read "Warning, Server1 is down." instead of "Warning, 123.123.12.103 is down."

I have tried extract which didn't work.

If anyone can offer a hand that would be swell. I am not even sure this watcher is right. I think I might need to add a timestamp to it. It might still not work right.

This is my heartbeat.yml file:

heartbeat.monitors:
- type: http

  # List or urls to query
  urls:
    - https://foobar.com
    - https://foobar.com
    - https://foobar.com
  schedule: '@every 10s'

- type: icmp
  tags: ["Server1"]
  hosts:
  - 123.123.12.103#These are servers so have to use IP address
  schedule: '@every 10s'

- type: icmp
  tags: ["Server2"]
  hosts:
  - 123.123.12.105
  schedule: '@every 10s'

And my watcher

{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
          "indices" : "heartbeat-*",
        "body": {
          "query": {
            "match": { "monitor.status" : "down"}
          }
        }
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "Warning, {{ctx.payload.hits.hits.0._source.resolve.ip}} is down."
      }
    }
  }
}

The output

   "actions": [
      {
        "id": "my-logging-action",
        "type": "logging",
        "status": "simulated",
        "logging": {
          "logged_text": "Warning, 123.123.12.105 is down."
        }
      }
    ]
  }

(Mohamed Salih) #2

I have managed to figure out this watcher.
I will post the JSON code below in case anyone needs it. Since I weren't able to find an example online when I was making it, it might be helpful for someone.

{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat-*"
        ],
        "types": [],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10s"
                    }
                  }
                },
                {
                  "match": {
                    "monitor.status": "down"
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
  "actions": {
    "notify-slack": {
      "throttle_period_in_millis": 2000,
      "slack": {
        "message": {
          "to": [
            "#watcher"
          ],
          "text": "Warning. Host: {{ctx.payload.hits.hits.0._source.tags.0}} is down"
        }
      }
    }
  }
}

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.