Elastic Machine Learning - Pre Built Rules

[geetika_gopi]

Hello,

For some of the windows pre-built ML jobs, we noticed the datafeed configured was as follows:

{
  "bool": {
    "filter": [
      {
        "term": {
          "destination.ip": "169.254.169.254"
        }
      }
    ]
  }
}

Now when I queried these manually on my historical winlogbeat logs, elastic returned no such events. Which means my ML job essentially has no datafeed (or sample data) to analyse/work on.

Subsequently we noticed the following job messages:

1. Datafeed has been retrieving no data for a while
2. Datafeed stopped
3. Job is closing

Does it just mean that my ML job has no input records to initialize. If that is the case, why does the job get closed? Shouldn't it just keep looking?

Apologies in advance if my understanding is incorrect. Appreciate the support.

[richcollier]

Perhaps @Craig_Chamberlain can verify here, but I'm sure that all pre-built jobs wouldn't have a filter for a specific IP address. Can you tell us what job this is (it's name)?

[Craig_Chamberlain]

This is one of the jobs that looks for anomalous / suspicious activity involving the metadata service in a cloud environment. The job is for virtual instances running in cloud environments and can be stopped if there are no cloud instances instrumented with endpoints or agents.

[geetika_gopi]

Thanks @Craig_Chamberlain,

Could you tell me about this job: v2_windows_anomalous_network_activity_ecs
If there are no historical events are found according to the configured datafeed filter, the ML job goes to closed state. How do we prevent this from happening?

[richcollier]

All that I can say is that if you look at how the original job is constructed, there is no such filter.

See the out-of-the-box datafeed configuration on Github:

elastic/kibana/blob/8.2/x-pack/plugins/ml/server/models/data_recognizer/modules/security_windows/ml/datafeed_v2_windows_anomalous_network_activity_ecs.json