How to stop sending duplicate Slack notifications for the same error?

[ddregalo]

I've asked this question in X-Pack +Heartbeat forums with no real progress made - I was referred to @michael.heldebrant for some assistance!?? If you have any insight on this it would be very much appreciated!

Here's the details...

I have configured a heartbeat watcher to action a slack notification if the monitor status is down in the production environment and this is working fine buuuuuuuut - what I would like to do is NOT send duplicate notifications for the same error. Hence, is it possible to check the condition that triggers the action with the previous result and NOT send if equal??

Here is an example of a watcher:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat*"
        ],
        "types": [],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "_index": "heartbeat*"
                  }
                }
              ],
              "filter": [
                {
                  "term": {
                    "monitor.status": "down"
                  }
                },
                {
                  "term": {
                    "fields.environment": "Production"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m",
                      "lt": "now"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "unique_hosts": {
              "terms": {
                "field": "monitor.host"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "notify-slack": {
      "throttle_period_in_millis": 1800000,
      "slack": {
        "message": {
          "to": [
            "#heartbeat-production"
          ],
          "text": "*SUMMARY:* Encountered {{ctx.payload.aggregations.unique_hosts.buckets.size}} unique hosts with status 'down' in the last 5 mins\n*ENVIRONMENT*: {{ctx.payload.hits.hits.0._source.fields.environment}}\n\n*URLs:*\n{{#ctx.payload.aggregations.unique_hosts.buckets}} Host Name: {{key}}\n{{/ctx.payload.aggregations.unique_hosts.buckets}}",
          "icon": "https://image.freepik.com/free-icon/letter-p_318-9235.jpg"
        }
      }
    }
  }
}

[andres-perez (Andrés Pérez)]

Can you set a longer throttle_period?
I see it set to 30 minutes in your example, maybe a bigger value would be enough.

I think it's the most suitable mechanism for handling duplicate notifications.
Maybe we can suggest to the elastic devs to implement some kind of "forever" value for that parameter

If that were not possible, what comes to my mind would be much more cumbersome:

• Add another search (so chain inputs) to obtain the results of the last execution (e.g. sort a small time range) recorded in the .watcher-history-... index each time the watcher is triggered.
• Add comparison logic (so probably a script comparison) to take that values into account.

You can find related information in this question:

Alert when okay

Hi @acchaulk, Watcher can absolutely do this because it is just a matter of defining a condition to trigger on. If the condition passes, such as when an "error" state is detected, then you can follow up by performing any action that is supported by Watcher. Therefore, the issue that you may be having is how to define the right condition and there are a lot of different strategies that you can employ, depending on the complexity that you are willing to endure: Create two Watches The first W…

[ddregalo]

Thanks for the response @andres-perez Your first suggestion of a longer throttle period sounds like a reasonable quick fix though I may have a go at your second...a little hacky but if it works would be optimal...cheers! -D