in one of our projects we want to monitor the components of several data processing pipelines.
Each of these pipelines has the following structure:
- Connects to at least one data source
- Connects to at least one data sink
- Consists of several applications which consume from / publish to one or multiple solace topics / queues
The monitoring shall be based on two data sources:
- A repository with the configuration data of the pipelines as they should be (ideal world)
- Application logs from each element with the real configuration (real world)
The goal is to visualize each pipeline as some kind of flow chart and also show the health of each element in the pipeline (if it is up and running or if it is down / broken). The visualization should be in a way that it can be embedded in a dashboard.
Is there a Kibana app or a plugin which is suitable for this use case?