Ingesting alarm lists from an external system

Greetings,

we are currently centralizing our logging and monitoring infrastructure around a central ELK stack. Processing log messages has been a breeze, of course, and we have also had a quick primer on how to tackle time-series data, but there's a third flavor of information that we would like to ingest: alert lists.

We have a few vendor-specific non-ELK systems that present a list of alarms; items disappear from this list when they are resolved. Our goal is to have the ELK stack collect and present such a list of issues, combining multiple sources:

  • alerts derived within the ELK stack from log messages ("login attempt on switch SW321 at 4am")
  • alerts derived within the ELK stack from time series data ("switch SW321 port 46 has been communicating with >75% of its link data rate over the last 5min")
  • alerts ingested from external alerting systems ("optical networking management system says multiplexer OM765 port 12 is down")

These external systems support access via a REST API with, conceptually, the following endpoints:

  • /alerts: list of IDs of currently active alerts
  • /alerts/<id> details for the alert with the given ID

As I understand it, the ELK stack is built more around events ("at 12:34pm, the connection on switch SW321 port 46 went down") than state ("the connection on switch SW321 port 46 is currently down"), which raises the question of how best to ingest such alert lists. Is there a structure that the community agrees on?

Thanks in advance and kind regards!

Welcome to the forum. You ask an interesting question.

Maybe be specific on these "external alerting systems", and it's possible there is some integration, or someone reading the thread with specific experience on those precise systems and how they were integrated. In particular how the correct and current state of an (external) alert is maintained.

Hi!

The current specific case is a Nokia WaveSuite instance, but I'm pretty sure it won't be the last, which is why I was a bit vague initially.

I'm fine with writing my own integration or similar middleware, but I'd like to know how the data should be structured to fit into the ELK stack's data model, and, as you have mentioned, how to react to state changes in these external alerts.

Hey @RavuAlHemio ,

I think Elastic is going to introduce the ability to integrate with external alerting systems not sure when.

They talked about here:

I think I'd approach this as a state synchronization problem rather than treating every API response as a new event.

For each external system, I'd keep one document per active alarm, using a stable ID that combines the source system and its alarm ID. That way, polling the API updates existing documents instead of creating duplicates.

The interesting part is when an alarm disappears from the list. I'd mark it as resolved rather than deleting it, and maybe keep a separate history of state changes if you need to investigate incidents later.

One thing I'd be particularly careful about is failed or incomplete API responses. If the external system times out or returns only part of its alert list, you definitely don't want Elasticsearch to interpret all the missing alarms as resolved.

Maybe a simple last_seen timestamp combined with a successful polling cycle would be enough to handle that, provided the API returns a complete snapshot.

I feel like this would also make it easier to integrate additional vendors later, since each connector would only need to translate its own alarm format into the same internal structure.

I'd also store a @timestamp of the last poll and use an upsert script so missing ids get a resolved flag instead of delete it keeps history and kibana can show open vs closed