# Creating job: anomaly in the event rate of beats

**URL:** <https://discuss.elastic.co/t/creating-job-anomaly-in-the-event-rate-of-beats/261889>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [January 22, 2021, 9:22am UTC](https://discuss.elastic.co/t/creating-job-anomaly-in-the-event-rate-of-beats/261889 "2021-01-22T09:22:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 22, 2021, 9:22am UTC](https://discuss.elastic.co/t/creating-job-anomaly-in-the-event-rate-of-beats/261889/1 "2021-01-22T09:22:19Z")

</div>

Hello,

I would like to create a job to detect when there is anomaly in the number of event send by beats, mostly to detect if one of the beat stop sending logs to my ELK cluster (and to detect the name of the machine where the beat stopped working).

so for each beat I created a multy metric job, for the field `low count(Event rate)` and `split field by agent.name`

so the job for each beat looks like that:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5fa05585ada86611bfdf51e47717519e04a984eb.png)

I would like to know if it's the best way to do that, or there is another way to do it without spliting by `user.name` or by creating one job for all the beats instead of one job for each beat

Best regards

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [January 22, 2021, 4:38pm UTC](https://discuss.elastic.co/t/creating-job-anomaly-in-the-event-rate-of-beats/261889/2 "2021-01-22T16:38:52Z")

</div>

Seems like the best (and most efficient) way would be to create an Advanced job where:

1. query the index pattern where all of the beats publish the data (i.e. `filebeat-*`)
2. include a `terms` aggregation (sized appropriately to return data for all agents) which aggregates the counts for every agent.name and have that be the datafeed (follow example here: [https://www.elastic.co/guide/en/machine-learning/current/ml-configuring-aggregation.html](https://www.elastic.co/guide/en/machine-learning/current/ml-configuring-aggregation.html))
3. do `low count(Event rate)` and `split field by agent.name` making sure that you use the name of the terms aggregation as the split field and you use `"summary_count_field_name": "doc_count"` because you're having the ML job process the output of aggregations

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 19, 2021, 4:39pm UTC](https://discuss.elastic.co/t/creating-job-anomaly-in-the-event-rate-of-beats/261889/3 "2021-02-19T16:39:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.