Duplicate notifcations getting created in Slack for one watcher alert trigger

[marvnz (Martin Arndt)]

I have created the following advanced watch, that should create one slack notification when triggered but it is creating 2 notifications every time it is triggered:

{

"trigger": {
"schedule": {
"interval": "15m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"metricbeat-*"
],
"types": ,
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"match": {
"fields.env": "prod"
}
},
{
"range": {
"@timestamp": {
"lte": "now",
"gte": "now-{{ctx.metadata.window}}"
}
}
}
]
}
},
"aggs": {
"average_swap": {
"avg": {
"field": "system.memory.swap.used.pct"
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.aggregations.average_swap.value": {
"gte": "{{ctx.metadata.threshold}}"
}
}
},
"actions": {
"notify-slack": {
"throttle_period_in_millis": 300000,
"slack": {
"account": "monitoring",
"message": {
"from": "ElasticSearch Watcher",
"to": [
"#monitoring",
"@everyone"
],
"text": "Elastic Metricbeat Server starting to swap",
"attachments": [
{
"color": "danger",
"title": "EC2 Instance using swap",
"text": " EC2 Instance is starting to use swap memory, currently percentage is {{ctx.payload.aggregations.average_swap.value}} "
}
]
}
}
}
},
"metadata": {
"threshold": 1,
"window": "15m"
}
}

As I am using an aggregation, I only get one value to run a condition on, so not sure why I am getting 2 notifications in Slack.

Any ideas?

[spinscale (Alexander Reelsen)]

which Elasticsearch version are you using?

[marvnz (Martin Arndt)]

We are running 6.5.0 in Elastic Cloud

[spinscale (Alexander Reelsen)]

can you include the execution time of the context in the text to check if these are really from one run?

like "text": " EC2 Instance is starting to use swap memory, currently percentage is {{ctx.payload.aggregations.average_swap.value}} :sweat_smile:" {{ctx.trigger.triggered_time}}

Can you also include the output of the watch history?

GET .watcher-history-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "watch_id": "my_watch"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ]
}

Thanks!

[marvnz (Martin Arndt)]

Watcher History index for that day has gone so can't show that: