# ML job rare function to detect anomaly on IP address will report IP on same network as anomaly

**URL:** <https://discuss.elastic.co/t/ml-job-rare-function-to-detect-anomaly-on-ip-address-will-report-ip-on-same-network-as-anomaly/246213>\
**Category:** Kibana\
**Tags:** elastic-stack-machine-learning\
**Created:** [August 25, 2020, 4:06am UTC](https://discuss.elastic.co/t/ml-job-rare-function-to-detect-anomaly-on-ip-address-will-report-ip-on-same-network-as-anomaly/246213 "2020-08-25T04:06:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nameisnotimportant](https://avatars.discourse-cdn.com/v4/letter/n/ea5d25/32.png) [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\
**Post date:** [August 25, 2020, 4:06am UTC](https://discuss.elastic.co/t/ml-job-rare-function-to-detect-anomaly-on-ip-address-will-report-ip-on-same-network-as-anomaly/246213/1 "2020-08-25T04:06:36Z")

</div>

Hi,  
Referring to [https://discuss.elastic.co/t/ml-how-to-find-anomaly-from-ip-address/239996](https://discuss.elastic.co/t/ml-how-to-find-anomaly-from-ip-address/239996)

I found that IP address on same network will be detected as anomaly.  
e.g. let say 10.180.1.161 is the regularly used IP address by a specific user, when that same user log in with IP address of 10.180.1.162, the ML job will report that as anomaly as well.

Is there a way to filter above scenario out from the result ? Is it possible to use custom rules to do that filter? If yes, may i have some sample code on how to do so?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 25, 2020, 2:51pm UTC](https://discuss.elastic.co/t/ml-job-rare-function-to-detect-anomaly-on-ip-address-will-report-ip-on-same-network-as-anomaly/246213/2 "2020-08-25T14:51:49Z")

</div>

What if, instead of analyzing the entire IP address, in the ML datafeed use a [`script_field`](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/search-fields.html#script-fields) to create just a subsection of the IP address (i.e. the first octet?)

So, instead of passing `109.180.1.161` to ML, you'd only be passing `109`. In that way, the rarity of the first octet per user should be more effective.

Example query:

```auto
GET yourindexname/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields": {
    "ip_first_octet": {
      "script": {
        "source": """
            def m = /^([0-9]+)\..*$/.matcher(doc['clientip'].value);
            if ( m.matches() ) {
              return Integer.parseInt(m.group(1))
            } else {
              return 0
            }
          """
      }
    }
  }
}

```

(obviously, above needs to be adapted to be incorporated into an ML datafeed query)

Also: note that in order to get the above to work, you might have to set `script.painless.regex.enabled: true` in elasticsearch.yml to allow regex matching

If this idea works effectively, consider doing the subsection at ingest time to avoid the overhead of calculating the `script_field` at query time.

---

<div class="post-metadata">

**Author:** ![nameisnotimportant](https://avatars.discourse-cdn.com/v4/letter/n/ea5d25/32.png) [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\
**Post date:** [August 27, 2020, 2:05am UTC](https://discuss.elastic.co/t/ml-job-rare-function-to-detect-anomaly-on-ip-address-will-report-ip-on-same-network-as-anomaly/246213/3 "2020-08-27T02:05:55Z")

</div>

Thanks richcollier, after few days of research, i finally see your point now. I will pre-process the IP address during ingest time to read just the 1st octet for IPv4.  
And for IPv6, can i just read in the first 3 blocks (the Global Unitcast Address)? I did tried to understand IPv6 structure from [here](http://www.steves-internet-guide.com/ipv6-guide/) but i think i still need more research to fully understand the structure.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 24, 2020, 2:06am UTC](https://discuss.elastic.co/t/ml-job-rare-function-to-detect-anomaly-on-ip-address-will-report-ip-on-same-network-as-anomaly/246213/4 "2020-09-24T02:06:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.