Elastic Machine learning jobs - Prometheus Blackbox Exporter / Kafka Metrics

[jk247]

Hi everyone,

I am currently setting up a proactive monitoring solution with the help of the Elastic Machine Learning feature for a Web application. (In order to see the benefits vs my current (reactive) setup with Grafana)
All of the ML jobs that I already set up are based on Elasticsearch Logs. In addition to that I have prometheus with Blackbox exporter set up to do http checks in order to check surrounding services for their availability.
Now to my question: Is it possible to analyze the prometheus metrics with the Elastic ML feature in order to implement a proactive monitoring solution? Like is it realistic that the algorithm can detect when a service might have a downtime or similar issues with this setup? Can anyone share their experiences? I would be really really thankful for that!

An additional scenario would be to analyze Kafka metrics that also come via Prometheus. Anyone has experiences with that / Can share useful use cases?

Thank you very much in advance for the help!

[richcollier]

Obviously, Elastic ML does real-time anomaly detection so it is perfectly capable of being part of your proactive monitoring goals. It can find anomalies in metric based data as well as log data (see some of the "built-in" ML jobs we distribute here: Supplied anomaly detection configurations | Machine Learning in the Elastic Stack [7.11] | Elastic)