How to stop sending duplicate slack notifications of the same error?


#1

I have configured a heartbeat watcher to action a slack notification if the monitor status is down in the production environment and this is working fine buuuuuuuut - what I would like to do is NOT send duplicate notifications for the same error. Hence, is it possible to check the condition that triggers the action with the previous result and NOT send if equal??

Here is an example of a watcher:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat*"
        ],
        "types": [],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "_index": "heartbeat*"
                  }
                }
              ],
              "filter": [
                {
                  "term": {
                    "monitor.status": "down"
                  }
                },
                {
                  "term": {
                    "fields.environment": "Production"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m",
                      "lt": "now"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "unique_hosts": {
              "terms": {
                "field": "monitor.host"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "notify-slack": {
      "throttle_period_in_millis": 1800000,
      "slack": {
        "message": {
          "to": [
            "#heartbeat-production"
          ],
          "text": "*SUMMARY:* Encountered {{ctx.payload.aggregations.unique_hosts.buckets.size}} unique hosts with status 'down' in the last 5 mins\n*ENVIRONMENT*: {{ctx.payload.hits.hits.0._source.fields.environment}}\n\n*URLs:*\n{{#ctx.payload.aggregations.unique_hosts.buckets}} Host Name: {{key}}\n{{/ctx.payload.aggregations.unique_hosts.buckets}}",
          "icon": "https://image.freepik.com/free-icon/letter-p_318-9235.jpg"
        }
      }
    }
  }
}


(Andrew Cholakian) #2

It might be best to ask this question in the watcher forum: https://discuss.elastic.co/c/x-pack/watcher

I agree that this is tricky at the moment, but it is a part of our stack we are working to improve.

I'm going to ping @michael.heldebrant who has some familiarity on this topic, he might be able to chime in.


#3

Thanks Andrew, just got back from a short leave and will follow-up with the watcher forum as well as Michael H.!


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.