ML: how to find anomaly from ip address

Elastic Stack Kibana
1

Hi,
I have been trying to figure out how to use the multi metric to find anomaly in IP address. Objective is to do something like gmail login notification if IP address or device type are different.

I created sample data from CSV with logindate, userid, ipaddress. In my data, the userid is always the same and logindate is always different. Out of 15,000 records, only 1 different ipaddress.

I created anomaly detection jobs by choosing count distinct userid as the field and ipaddress as the influencer. And there is no anomaly found. I tried to switch the two as in count distinct ipaddress as field and userid as the influencer, still no anomaly found.

May i know what is the problem? My data? or the way i pick the field and influencer?

2

The way to do something like this would be using the advanced job wizard. Here, you can select to partition by user and then look for rare items by ip address and select influencer as user. This will tell you about unusual ip addresses that a given userid uses. Depending on cardinality of fields this may be effective, but it assumes that you have a stable population userid and see reasonably frequent logins from each one. Note as well that rare functions assume that the data are spread over time, they actually tell you about things which occur only infrequently among the time buckets the job has analysed. If you put all your data in the same time bucket it won't detect anything.

3

Thanks Tom. Let me try it out.
Update: I just got the result i need. Thanks a lot.

4

There are three more steps i need help with for this task. The requirement is to prompt notification. I am from reporting background where i use only SQL (non programming base), so i am not sure how to do this programmatically.

But i assume i need to make it a real time job and convert the ML job into codes that can be executed using curl and then make it a 'function' like in SQL so that when the front end call that 'function', it will return the alert message.

May i ask for the direction, or the key words that i need to google for to complete the three steps? Front end using dot net codes.

5

Alerting on ML jobs currently relies on the Watcher tool that is built into the Elastic Stack.

Although a little dated, this blog describes the fundamentals of using Watcher to alert on ML jobs: https://www.elastic.co/blog/alerting-on-machine-learning-jobs-in-elasticsearch-v55

6

richcollier suggestion is also one of the solution to my questions. however, this page only allow one solution. So, whoever read on this thread, check out the alerting feature also.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.