How to start a machine learning job to check if a user starts an application they normally do not use with Kibana

Elastic Stack Kibana
1

I am trying to set up a machine learning job to detect when non admin users who normally do not use powershell start powershell.

I already saved a search function that shows powershell started logs, and filters the unimportant users:

event.action: "created-process" and process.name: "powershell.exe" and not related.user: (*$ or adm_*)

Now I thought the way to go is to use the data from the saved search in a machine learning job running a rare detector with related.user as the field. My thinking is that this way it finds when powershell was started, and registers an anomaly when someone runs it who does not usually use powershell. However, searching for rare by "related.user" does not seem to be an option in Elasticsearch. Does anyone have an example of how I could set this up?

2

What is the mapping of the related.user field? Is it a keyword field?

3

My apologies, I am quite new to Elasticsearch and kibana. How can I check this?

4

You can use the Elasticsearch Get Mapping API: Get mapping API | Elasticsearch Guide [8.2] | Elastic

It's easiest to call if you go to Dev console in Kibana, which you can get to by clicking "View in Console" on one of the examples on that docs page.

Things might be more complicated if you are searching across many indices and they have different mappings for this field. Which indices is your saved search configured to search?

5

I believe this is the only one:
"indices": [
"winlogbeat-*"
],

6

related.user does not appear in this file. Neither do host.name or user.name. Only username and hostname. but there is no entry for related.user or relateduser.

7

I think the misunderstanding here is that you do not search for rare by related.user in the actual Elasticsearch query language, you accomplish that bit using an ML job (see rarity analysis article)

So, you need to:

  1. Create a filtered search to come up with a version of the data set that you want - it seems that you've done this part. Save this search as a "Saved Search"
  2. Use that "Saved Search" as the basis of your ML job
  3. Configure your ML job to do rarity analysis using the appropriate fields in the data.

I hope this helps

8

Thats what i am trying, but when i set up an advanced machine learning job related.user does not appear in the field.

image
image1459×692 95 KB

I tried using user.name.text as alternative:

image
image1630×504 92.8 KB

The datafeed preview shows correct data but i still cant start the job:

image
image1627×299 22.1 KB

9

The UI is suggesting you use related.user.keyword (the keyword type version of related.user). This will work for you.

If you did not know this, the index "mappings" are definitions of the fields and their types. A long time ago, string-based fields (like fields that defined the names of things) were replaced with text and keyword types.

10

I tried that, but it shows 0 related users. Only timestamps:

image
image1209×332 60.7 KB

And when i look for a user that definitely exists it cannot be found:

image
image1162×256 13.3 KB

I aslo tried using the user.name.text option which does show the correct usernames in the data preview, but apparently i cant use it:

image
image805×276 12.7 KB

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.