Aggregation of single jobs in machine learning

[Emmanuel (Bulard)]

I would like to aggregate or to consolidate different single jobs or multiple jobs in order to correlate different issues I could have for one application.
What I would like to do is:

• One multiple jobs based on performance metrics for 5 servers of my application (Inside the index metricbeat)
• One multiple jobs based on error in the logs for 5 servers of my applications (Inside the index filebeat)
• One single job based on the usage of my applications (Inside the index filebeat)

The objective is to have alert just only if I have at least two issues (one related to usage and one related to one server capacity).
How can I consolidate different jobs to send notification just if I have different issues correlated?

[droberts195 (David Roberts)]

Hi,

This is the sort of functionality we're looking to make much more accessible in a future release.

It is possible today if you're prepared to create some complicated watches, but in the future we'll add a wrapper so that you don't have to create the watch directly.

Here are some clues if you want to try and use watcher with ML in 5.4:

• By default ML results are stored in an index called .ml-anomalies-shared, so unless you changed this then the results from all 3 of your jobs will be in there
• The results have a field job_id which indicates which job they came from, plus a timestamp that indicates which bucketed time period they relate to
• There are different types of results, indicated by the result_type field
• If you want to alert when any two jobs have overall anomalies at the same time then you want to look for the anomaly_score field being greater than some threshold (say 75) when result_type is bucket, and alert when there are two results found with the same or similar timestamps

Basically it is possible, but unless you're a watcher expert you may prefer to wait until we add functionality to automate the watch setup.