Our organization is trying to create a scalable framework for how any alert built by watcher analysis logic will (a) have a configuration metdata setup for each watcher and (b) have a well-formed output index structure that is suited for exposing all our alerts nicely on a professional business portal in kibana for exploration and dashboard 'heat' analysis for business operations staff.
Watcher that we did so far is a bit of a hack. Spaghetti code at best with things like distribution lists embedded in code done for the alerts, no functional encapsulation of key modules that build watchers.. basically a bunch of watchers all repeating same logic over and over again.. quite a mess.
Has anyone built a framework on watcher for configurating/storing properties for all your watchers and also encapsulating logic in a scalable way ?
Most vendors seem to do a nice job of encapsulating a service ( watcher ) with helper UI admin config aids and standard structures for setting things up in a clean/scalable way like the distribution list topic above... but we see nothing.
Thoughts ? We are concerned really that watcher doesn't do that much in a sophisticated way and that we already have alot of poor formed logic in watcher that doesn't scale well to build more and more alerts in watcher.
Any advice ? Places to look ?