Greetings,
we are currently centralizing our logging and monitoring infrastructure around a central ELK stack. Processing log messages has been a breeze, of course, and we have also had a quick primer on how to tackle time-series data, but there's a third flavor of information that we would like to ingest: alert lists.
We have a few vendor-specific non-ELK systems that present a list of alarms; items disappear from this list when they are resolved. Our goal is to have the ELK stack collect and present such a list of issues, combining multiple sources:
- alerts derived within the ELK stack from log messages ("login attempt on switch SW321 at 4am")
- alerts derived within the ELK stack from time series data ("switch SW321 port 46 has been communicating with >75% of its link data rate over the last 5min")
- alerts ingested from external alerting systems ("optical networking management system says multiplexer OM765 port 12 is down")
These external systems support access via a REST API with, conceptually, the following endpoints:
/alerts: list of IDs of currently active alerts/alerts/<id>details for the alert with the given ID
As I understand it, the ELK stack is built more around events ("at 12:34pm, the connection on switch SW321 port 46 went down") than state ("the connection on switch SW321 port 46 is currently down"), which raises the question of how best to ingest such alert lists. Is there a structure that the community agrees on?
Thanks in advance and kind regards!