Creating job: anomaly in the event rate of beats

[TheHunter1]

Hello,

I would like to create a job to detect when there is anomaly in the number of event send by beats, mostly to detect if one of the beat stop sending logs to my ELK cluster (and to detect the name of the machine where the beat stopped working).

so for each beat I created a multy metric job, for the field low count(Event rate) and split field by agent.name

so the job for each beat looks like that:

I would like to know if it's the best way to do that, or there is another way to do it without spliting by user.name or by creating one job for all the beats instead of one job for each beat

Best regards

[richcollier]

Seems like the best (and most efficient) way would be to create an Advanced job where:

1. query the index pattern where all of the beats publish the data (i.e. filebeat-*)
2. include a terms aggregation (sized appropriately to return data for all agents) which aggregates the counts for every agent.name and have that be the datafeed (follow example here: https://www.elastic.co/guide/en/machine-learning/current/ml-configuring-aggregation.html)
3. do low count(Event rate) and split field by agent.name making sure that you use the name of the terms aggregation as the split field and you use "summary_count_field_name": "doc_count" because you're having the ML job process the output of aggregations