We are Alerting our customers when there is a problem in their infrastructure,Basically we are using ELK as a Monitoring solution.
We are having around 300+ watcher to notify when something goes wrong based on different alert type and condition ,now the problem is in weekend server bounces,upgrading like this other activities are going on. At this time we don't want to alert our customers false positive alerts because everybody knows that some activities are happening at this time.
To avoid sending False Positive alerts we are disabling watchers until activities gets completed .
Every time we are enabling and disabling all watchers manually which is very pathetic.
What is the best way to automate watchers(Enable/Disable) in ELK,Is there any specific API's or can we schedule a query to perform this?
Seeking your help!!