# Anomaly detector query to find IPs that only visit a specific page?

**URL:** <https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [August 22, 2022, 6:05pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647 "2022-08-22T18:05:09Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 22, 2022, 6:05pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/1 "2022-08-22T18:05:09Z")

</div>

I am looking for a way to find IPs that are only hitting a specific page repeatedly to detect credential stuffing with an anomaly detector.

I currently have a detector setup to look for 401 responses coming from visits to the page, but unfortunately it is not fine-toothed enough.

I've been trying to find a way to only filter for IPs that are visiting _only_ this specific page and none others.

Does anyone know of a query that can accomplish this?

Thanks!

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 24, 2022, 1:06am UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/2 "2022-08-24T01:06:04Z")

</div>

It's going to be something like this, which uses a `bucket_selector` aggregation:

```auto
GET kibana_sample_data_logs/_search
{
  "size": 0,
  "aggs": {
    "ips": {
      "terms": {
        "field": "clientip",
        "size": 10000
      },
      "aggs": {
        "not_target_url": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "must_not": {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "request": "/elasticsearch/elasticsearch-6.3.2.deb"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    }
                  }
                }
              ]
            }
          }
        },
        "target_url": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "match_phrase": {
                          "request": "/elasticsearch/elasticsearch-6.3.2.deb"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        },
        "expose_bots": {
          "bucket_selector": {
            "buckets_path": {
              "target_url": "target_url._count",
              "not_target_url": "not_target_url._count"
            },
            "script": "params.target_url > 500 && params.not_target_url == 0"
          }
        }
      }
    }
  }
}

```

First, you `terms` agg on clientip, then do two `filter` aggregations, one to select the `request` URL you don't want to include, and then one which you DO want to include. Then, use the `bucket_selector` aggregation to only expose those IPs that have a count of the target URL \> 500 hits (or whatever arbitrary number you choose) but no counts (no hits) of any other URL.

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 24, 2022, 3:37pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/4 "2022-08-24T15:37:58Z")

</div>

Thanks for this!

I tried to add in a bit more filtering and renamed things to be clearer, then pointed it at an index from a day where I know we had a bot, but did not get any hits. Does all this look right?

```auto
GET access-logs-<date>/_search
{
  "size": 0,
  "aggs": {
    "ips": {
      "terms": {
        "field": "clientip.keyword",
        "size": 10000
      },
      "aggs": {
        "not_login_url": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "must": {
                      "match": {
                        "app": "<appName>"
                      }
                    },
                    "must_not": {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "request": "/account"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    }
                  }
                }
              ]
            }
          }
        },
        "login_url": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "must": {
                      "match": {
                        "app": "<appName>"
                      }
                    },
                    "should": [
                      {
                        "match_phrase": {
                          "request": "/login"
                        }
                      },
                      {
                        "match": {
                          "response": 401
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        },
        "expose_bots": {
          "bucket_selector": {
            "buckets_path": {
              "login_url": "login_url._count",
              "not_login_url": "not_login_url._count"
            },
            "script": "params.login_url > 10 && params.not_login_url == 0"
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 24, 2022, 4:28pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/5 "2022-08-24T16:28:44Z")

</div>

Hard to debug just by looking at query text. You should debug by making a separate query for each of the two cases. For example:

```auto
GET access-logs-<date>/_search
{
    "query": {
    "bool": {
      "filter": [
        {
               {
                  "bool": {
                    "must": {
                      "match": {
                        "app": "<appName>"
                      }
                    },
...
}
 

```

and inspect the output to see if it is giving you what you want

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 24, 2022, 6:36pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/7 "2022-08-24T18:36:06Z")

</div>

So when I ran these two separate queries, the results looked good.

```auto
# Failed logins
GET <shard>/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "request": "/login"
                }
              },
              {
                "match": {
                  "response": 401
                }
              }
            ],
            "minimum_should_match": "100%"
          }
        }
      ]
    }
  }
}

```

```auto
# All other requests to the app
GET <shard>/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": {
              "match": {
                "app": "appName"  
              }
            },
            "must_not": {
              "bool": {
                "should": [
                  {
                    "match_phrase": {
                      "request": "/login"
                    }
                  }
                ],
                "minimum_should_match": "100%"
              }
            }
          }
        }
      ]
    }
  }
}

```

I'm wondering if the script portion when running them in the aggregation query is the problem?

Here is the whole thing based off your example:

```auto
GET <shard>/_search
{
  "size": 0,
  "aggs": {
    "ips": {
      "terms": {
        "field": "clientip.keyword",
        "size": 10000
      },
      "aggs": {
        "not_login_url": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "must": {
                      "match": {
                        "app": "appName"
                      }
                    },
                    "must_not": {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "request": "/login"
                            }
                          }
                        ],
                        "minimum_should_match": "100%"
                      }
                    }
                  }
                }
              ]
            }
          }
        },
        "login_url": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "match_phrase": {
                          "request": "/login"
                        }
                      },
                      {
                        "match": {
                          "response": 401
                        }
                      }
                    ],
                    "minimum_should_match": "100%"
                  }
                }
              ]
            }
          }
        },
        "bots": {
          "bucket_selector": {
            "buckets_path": {
              "login_url": "login_url._count",
              "not_login_url": "not_login_url._count"
            },
            "script": "params.login_url > 10 && params.not_login_url == 0"
          }
        }
      }
    }
  }
}

```

When I removed the `params.not_login_url == 0` portion of the script, I got no output:

```auto
"hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : []
  },
  "aggregations" : {
    "ips" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 4368487,
      "buckets" : []
    }
  }

```

When I removed the `params.login_url > 10` portion, I got an output of lots of buckets, but they all look like

```auto
"key" : "<IP>",
          "doc_count" : <someNumber>,
          "login_url" : {
            "doc_count" : 0
          },
          "not_login_url" : {
            "doc_count" : 0
          }

```

So for some reason `params.not_login_url` is never 0, and `params.login_url` is frequently above 10, but the separate aggregations are not working.

Any ideas?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 25, 2022, 6:05pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/8 "2022-08-25T18:05:46Z")

</div>

The logic of the script says that there needs to be more than 10 occurrences of a single IP hitting the login page, and exactly zero occurrences of that same IP hitting any other page. So, is it just possible that in your case, over the dataset that you are looking, that no IPs match both those conditions at the same time?

For example, an IP address that hits the login page 500 times, but also has just a single hit on any other page will NOT be selected with this logic.

Keep in mind that the `bucket_selector` aggregation is like a filter. Whatever matches in the script is what you'll get. So, when you remove the `params.login_url > 10` portion, you're leaving in the `params.not_login_url == 0` portion - so yes, all of the output buckets will be where `not_login_url==0`:

```auto
          "not_login_url" : {
            "doc_count" : 0
          }

```

So when you say:

> So for some reason `params.not_login_url` is never 0

That's not true because you just showed examples of where it is zero.

So, you have to prove to me first that you have an IP address in your data set that meets both conditions (hits the login more than 10 times but hits no other url even once) - because now I'm not convinced that you do

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 25, 2022, 6:19pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/9 "2022-08-25T18:19:58Z")

</div>

Hi Rich,  
I manually verified that in the index I was looking at there was no instance of failed logins without visiting other pages.

I then connected to a VPN and checked to make sure that there were no logs in the index with that IP address.

Finally, I sent 20 curl requests that resulted in 401s to the endpoint and confirmed it was visible in the logs.

So I know for certain that there is exactly 1 occurrence of this in the index I am searching.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\
**Post date:** [August 25, 2022, 8:07pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/10 "2022-08-25T20:07:22Z")

</div>

Check out the significant\_terms agg. It is designed to find terms (in your case, users) that are almost exclusively related to a given query (in your case, the questionable page).

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 25, 2022, 9:13pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/11 "2022-08-25T21:13:54Z")

</div>

Hi Mark,  
That works well for the failed logins query on its own

```auto
GET <shard>/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "request": "/login"
                }
              },
              {
                "match": {
                  "response": 401
                }
              }
            ],
            "minimum_should_match": "100%"
          }
        }
      ]
    }
  },
  "aggs": {
    "significant_ips": {
      "significant_terms": {
        "field": "clientip.keyword"
      }
    }
  }
}

```

But is there a way to combine this with a query that searches for the same IP hitting another URL besides the login?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\
**Post date:** [August 26, 2022, 7:07am UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/12 "2022-08-26T07:07:40Z")

</div>

Glad to hear it’s working for you.  
The request shown above will return multiple IPs not just one.  
If you are wanting to validate how many non-login accesses each ip has done the bg\_count part of the response should help with that.  
However if you are wanting to see what else these bad guys are attempting to access that differs from the good guys visiting your site, I demo that at 1h16m into [this video](https://www.elastic.co/elasticon/conf/2016/sf/opening-keynote). This is using the graph api in the backend to “walk out” the connections between visitors and urls. You can use that api or write custom code in your client to mimic the behaviour with multiple search calls.

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 26, 2022, 2:20pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/13 "2022-08-26T14:20:47Z")

</div>

Oh okay, so if I'm reading this correctly, `doc_count` is the number of `/login` hits, and `bg_count` is the total number of pages hit by the IP. So if I wrote a script that says `(bg_count - doc_count) == 0` then the query should only return buckets with IPs that hit the `/login` url and nothing else, correct?

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\
**Post date:** [August 26, 2022, 4:35pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/14 "2022-08-26T16:35:41Z")

</div>

Correct but see the caveats about the accuracy in the significant terms docs.  
Also bg\_count is the number of accesses (docs) not the number of unique pages accessed.

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 26, 2022, 6:27pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/15 "2022-08-26T18:27:55Z")

</div>

Good to know, thank you.

Is there a way that I can add a script to the aggregation to only return buckets that have a ratio of failed logins / bg \>= some number?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\
**Post date:** [August 27, 2022, 2:35pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/16 "2022-08-27T14:35:16Z")

</div>

Significant terms agg can take a scripted scoring heuristic which you could make return zero when appropriate.

---

<div class="post-metadata">

**Author:** ![parkerjthomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parkerjthomas/32/109940_2.png) [@parkerjthomas](https://discuss.elastic.co/u/parkerjthomas)\
**Post date:** [August 29, 2022, 1:59pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/17 "2022-08-29T13:59:53Z")

</div>

Thanks for all the help. I found that the `percentage {}` field worked well enough for my needs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 26, 2022, 2:00pm UTC](https://discuss.elastic.co/t/anomaly-detector-query-to-find-ips-that-only-visit-a-specific-page/312647/18 "2022-09-26T14:00:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.