How to start a machine learning job to check if a user starts an application they normally do not use with Kibana

[anddam (Andreas Damen)]

I am trying to set up a machine learning job to detect when non admin users who normally do not use powershell start powershell.

I already saved a search function that shows powershell started logs, and filters the unimportant users:

event.action: "created-process" and process.name: "powershell.exe" and not related.user: (*$ or adm_*)

Now I thought the way to go is to use the data from the saved search in a machine learning job running a rare detector with related.user as the field. My thinking is that this way it finds when powershell was started, and registers an anomaly when someone runs it who does not usually use powershell. However, searching for rare by "related.user" does not seem to be an option in Elasticsearch. Does anyone have an example of how I could set this up?

[droberts195 (David Roberts)]

What is the mapping of the related.user field? Is it a keyword field?

[anddam (Andreas Damen)]

My apologies, I am quite new to Elasticsearch and kibana. How can I check this?

[droberts195 (David Roberts)]

You can use the Elasticsearch Get Mapping API: Get mapping API | Elasticsearch Guide [8.2] | Elastic

It's easiest to call if you go to Dev console in Kibana, which you can get to by clicking "View in Console" on one of the examples on that docs page.

Things might be more complicated if you are searching across many indices and they have different mappings for this field. Which indices is your saved search configured to search?

[anddam (Andreas Damen)]

I believe this is the only one:
"indices": [
"winlogbeat-*"
],

[anddam (Andreas Damen)]

@droberts195

related.user

related.user does not appear in this file. Neither do host.name or user.name. Only username and hostname. but there is no entry for related.user or relateduser.

[richcollier]

I think the misunderstanding here is that you do not search for rare by related.user in the actual Elasticsearch query language, you accomplish that bit using an ML job (see rarity analysis article)

So, you need to:

1. Create a filtered search to come up with a version of the data set that you want - it seems that you've done this part. Save this search as a "Saved Search"
2. Use that "Saved Search" as the basis of your ML job
3. Configure your ML job to do rarity analysis using the appropriate fields in the data.

I hope this helps

[anddam (Andreas Damen)]

Thats what i am trying, but when i set up an advanced machine learning job related.user does not appear in the field.

I tried using user.name.text as alternative:

The datafeed preview shows correct data but i still cant start the job:

[richcollier]

The UI is suggesting you use related.user.keyword (the keyword type version of related.user). This will work for you.

If you did not know this, the index "mappings" are definitions of the fields and their types. A long time ago, string-based fields (like fields that defined the names of things) were replaced with text and keyword types.

[anddam (Andreas Damen)]

I tried that, but it shows 0 related users. Only timestamps:

And when i look for a user that definitely exists it cannot be found:

I aslo tried using the user.name.text option which does show the correct usernames in the data preview, but apparently i cant use it: