# Monitoring Machine Learning Job States

**URL:** <https://discuss.elastic.co/t/monitoring-machine-learning-job-states/267587>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [March 18, 2021, 1:53am UTC](https://discuss.elastic.co/t/monitoring-machine-learning-job-states/267587 "2021-03-18T01:53:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![saraKM](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@saraKM](https://discuss.elastic.co/u/saraKM)\
**Post date:** [March 18, 2021, 1:53am UTC](https://discuss.elastic.co/t/monitoring-machine-learning-job-states/267587/1 "2021-03-18T01:53:30Z")

</div>

Hi,

I wonder what would be the best approach to get alerts when a job fails or goes into an unexpected state. I have some failed jobs and some jobs showing closed with started datafeed ( so moving from open/started to closed/started) - is there a way to get alerts on those job states. The notification index might help, but still very noisy with manual closing/starting of jobs.

Thanks,  
Sara

---

<div class="post-metadata">

**Author:** ![darnautov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darnautov/32/70608_2.png) [@darnautov](https://discuss.elastic.co/u/darnautov)\
**Post date:** [March 18, 2021, 1:32pm UTC](https://discuss.elastic.co/t/monitoring-machine-learning-job-states/267587/2 "2021-03-18T13:32:06Z")

</div>

Hi @saraKM,

At the moment you can utilize [Watcher](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/xpack-alerting.html) to get notified about the job and datafeed states.

Use a [search input](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/input-search.html) with querying the `.ml_notifications-*` index for messages your are interested in. Example of the input:

```auto
{
  "input": {
    "search": {
      "request": {
        "indices": [
          ".ml-notifications-*"
        ],
        "body": {
          "sort": {
            "timestamp": {
              "order": "desc"
            }
          },
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "timestamp": {
                      "gte": "now-15m"
                    }
                  }
                },
                {
                  "terms": {
                    "message.raw": [
                      "Datafeed stopped"
                    ]
                  }
                },
                {
                  "term": {
                    "job_id": {
                      "value": "my_job"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  }
}

```

To avoid extra noise consider action [throttling](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/actions.html#actions-ack-throttle).

FYI [Kibana Alerting](https://www.elastic.co/guide/en/kibana/7.11/alerting-getting-started.html) provides a more convenient way of managing your alerting rules and action. It's actively developing and new alert types are included with each release. Feel free to [create a GitHub issue](https://github.com/elastic/kibana/issues/new?assignees=&labels=:ml&template=Feature_request.md) in the Kibana repo for the alert type you're interested in.

Hope it helps.

Regards,  
Dima

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 15, 2021, 1:32pm UTC](https://discuss.elastic.co/t/monitoring-machine-learning-job-states/267587/3 "2021-04-15T13:32:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.