Cant' import Machine Learning Anomaly Jobs - I have Enterprise Cloud Subscription

[opiedrah (Elasticsearch Certified Professional)]

Hi,
We are an Elastic Cloud Enterprise customer. The machine learning jobs that are to be prebuilt are missing.

I have a ML node assigned to the cluster, i can't find where to import them.

Cluster is running 7.10.2

Thank you

[stephenb (Stephen Brown)]

Hi @opiedrah

So if this is a brand new cluster there will be no jobs until

a) You ingest data via an integration or a beat they will be created during the setup process example Metrics, APM etc...

b) You manually create one

Which Job were you expecting to see?

[richcollier]

There are indeed some "pre-built" jobs for the SIEM app and APM, but you need to enable/import those jobs from those apps - not from the general ML UI.

For example, reference: https://www.elastic.co/guide/en/siem/guide/current/machine-learning.html

[opiedrah (Elasticsearch Certified Professional)]

When i click under SIEM, ML job Settings it shows everything failed, and when i click on each one it shows no job.

No Job

[richcollier]

The error message indicates that there is no data in your cluster that these ML jobs require in order to install and run. Refer to the job definitions which show the required data necessary: https://www.elastic.co/guide/en/siem/guide/current/prebuilt-ml-jobs.html

For example, you can see that the linux_anomalous_network_url_activity_ecs requires data collected by from Auditbeat (Linux). And, as you can imagine, the ML jobs whose names begin with windows requires data collected via winlogbeat.

Unless you've collected that kind of data using those beats, the product is working as expected by not allowing you to enable jobs on data that doesn't yet exist.

[opiedrah (Elasticsearch Certified Professional)]

@richcollier

efer to the job definitions which show the required data necessa

Interesting... We are currently using fleet with the endpoint security integration, this captures security events , network, process etc. So what you are saying is that the ML jobs currently do not work with data from fleet+ElasticEndpoint. One would also have to install winlogbeat for the ML jobs to work?

Thanks

[richcollier]

Thank you for the additional information. Yes, there was a compatibility issue with built-in ML jobs not recognizing the indices if the data was collected via the new, consolidated Elastic Agent (since the names of the indices are different than if collected with legacy Beats). Word is that this is fixed in v7.11 due out next week.

[opiedrah (Elasticsearch Certified Professional)]

Thank you @richcollier I'll mark it as resolved and hope for the best for 7.11 or 7.12. Do you have any insights on this, or know someone that does?

Filebeat o365 module - Parsing o365.audit.data field

I think that using a script processor for this fields is not a bad idea. You can give a try to JSON processor (ingest node): https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html

Thanks again