I am looking for a way to find IPs that are only hitting a specific page repeatedly to detect credential stuffing with an anomaly detector.
I currently have a detector setup to look for 401 responses coming from visits to the page, but unfortunately it is not fine-toothed enough.
I've been trying to find a way to only filter for IPs that are visiting only this specific page and none others.
Does anyone know of a query that can accomplish this?
Thanks!
It's going to be something like this, which uses a bucket_selector aggregation:
GET kibana_sample_data_logs/_search
{
"size": 0,
"aggs": {
"ips": {
"terms": {
"field": "clientip",
"size": 10000
},
"aggs": {
"not_target_url": {
"filter": {
"bool": {
"filter": [
{
"bool": {
"must_not": {
"bool": {
"should": [
{
"match_phrase": {
"request": "/elasticsearch/elasticsearch-6.3.2.deb"
}
}
],
"minimum_should_match": 1
}
}
}
}
]
}
}
},
"target_url": {
"filter": {
"bool": {
"filter": [
{
"bool": {
"should": [
{
"match_phrase": {
"request": "/elasticsearch/elasticsearch-6.3.2.deb"
}
}
],
"minimum_should_match": 1
}
}
]
}
}
},
"expose_bots": {
"bucket_selector": {
"buckets_path": {
"target_url": "target_url._count",
"not_target_url": "not_target_url._count"
},
"script": "params.target_url > 500 && params.not_target_url == 0"
}
}
}
}
}
}
First, you terms agg on clientip, then do two filter aggregations, one to select the request URL you don't want to include, and then one which you DO want to include. Then, use the bucket_selector aggregation to only expose those IPs that have a count of the target URL > 500 hits (or whatever arbitrary number you choose) but no counts (no hits) of any other URL.
Thanks for this!
I tried to add in a bit more filtering and renamed things to be clearer, then pointed it at an index from a day where I know we had a bot, but did not get any hits. Does all this look right?
GET access-logs-<date>/_search
{
"size": 0,
"aggs": {
"ips": {
"terms": {
"field": "clientip.keyword",
"size": 10000
},
"aggs": {
"not_login_url": {
"filter": {
"bool": {
"filter": [
{
"bool": {
"must": {
"match": {
"app": "<appName>"
}
},
"must_not": {
"bool": {
"should": [
{
"match_phrase": {
"request": "/account"
}
}
],
"minimum_should_match": 1
}
}
}
}
]
}
}
},
"login_url": {
"filter": {
"bool": {
"filter": [
{
"bool": {
"must": {
"match": {
"app": "<appName>"
}
},
"should": [
{
"match_phrase": {
"request": "/login"
}
},
{
"match": {
"response": 401
}
}
],
"minimum_should_match": 1
}
}
]
}
}
},
"expose_bots": {
"bucket_selector": {
"buckets_path": {
"login_url": "login_url._count",
"not_login_url": "not_login_url._count"
},
"script": "params.login_url > 10 && params.not_login_url == 0"
}
}
}
}
}
}
Hard to debug just by looking at query text. You should debug by making a separate query for each of the two cases. For example:
GET access-logs-<date>/_search
{
"query": {
"bool": {
"filter": [
{
{
"bool": {
"must": {
"match": {
"app": "<appName>"
}
},
...
}
and inspect the output to see if it is giving you what you want
So when I ran these two separate queries, the results looked good.
# Failed logins
GET <shard>/_search
{
"query": {
"bool": {
"filter": [
{
"bool": {
"should": [
{
"match_phrase": {
"request": "/login"
}
},
{
"match": {
"response": 401
}
}
],
"minimum_should_match": "100%"
}
}
]
}
}
}
# All other requests to the app
GET <shard>/_search
{
"query": {
"bool": {
"filter": [
{
"bool": {
"must": {
"match": {
"app": "appName"
}
},
"must_not": {
"bool": {
"should": [
{
"match_phrase": {
"request": "/login"
}
}
],
"minimum_should_match": "100%"
}
}
}
}
]
}
}
}
I'm wondering if the script portion when running them in the aggregation query is the problem?
Here is the whole thing based off your example:
GET <shard>/_search
{
"size": 0,
"aggs": {
"ips": {
"terms": {
"field": "clientip.keyword",
"size": 10000
},
"aggs": {
"not_login_url": {
"filter": {
"bool": {
"filter": [
{
"bool": {
"must": {
"match": {
"app": "appName"
}
},
"must_not": {
"bool": {
"should": [
{
"match_phrase": {
"request": "/login"
}
}
],
"minimum_should_match": "100%"
}
}
}
}
]
}
}
},
"login_url": {
"filter": {
"bool": {
"filter": [
{
"bool": {
"should": [
{
"match_phrase": {
"request": "/login"
}
},
{
"match": {
"response": 401
}
}
],
"minimum_should_match": "100%"
}
}
]
}
}
},
"bots": {
"bucket_selector": {
"buckets_path": {
"login_url": "login_url._count",
"not_login_url": "not_login_url._count"
},
"script": "params.login_url > 10 && params.not_login_url == 0"
}
}
}
}
}
}
When I removed the params.not_login_url == 0 portion of the script, I got no output:
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"ips" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 4368487,
"buckets" : [ ]
}
}
When I removed the params.login_url > 10 portion, I got an output of lots of buckets, but they all look like
"key" : "<IP>",
"doc_count" : <someNumber>,
"login_url" : {
"doc_count" : 0
},
"not_login_url" : {
"doc_count" : 0
}
So for some reason params.not_login_url is never 0, and params.login_url is frequently above 10, but the separate aggregations are not working.
Any ideas?
The logic of the script says that there needs to be more than 10 occurrences of a single IP hitting the login page, and exactly zero occurrences of that same IP hitting any other page. So, is it just possible that in your case, over the dataset that you are looking, that no IPs match both those conditions at the same time?
For example, an IP address that hits the login page 500 times, but also has just a single hit on any other page will NOT be selected with this logic.
Keep in mind that the bucket_selector aggregation is like a filter. Whatever matches in the script is what you'll get. So, when you remove the params.login_url > 10 portion, you're leaving in the params.not_login_url == 0 portion - so yes, all of the output buckets will be where not_login_url==0:
"not_login_url" : {
"doc_count" : 0
}
So when you say:
So for some reason
params.not_login_urlis never 0
That's not true because you just showed examples of where it is zero.
So, you have to prove to me first that you have an IP address in your data set that meets both conditions (hits the login more than 10 times but hits no other url even once) - because now I'm not convinced that you do
Hi Rich,
I manually verified that in the index I was looking at there was no instance of failed logins without visiting other pages.
I then connected to a VPN and checked to make sure that there were no logs in the index with that IP address.
Finally, I sent 20 curl requests that resulted in 401s to the endpoint and confirmed it was visible in the logs.
So I know for certain that there is exactly 1 occurrence of this in the index I am searching.
Check out the significant_terms agg. It is designed to find terms (in your case, users) that are almost exclusively related to a given query (in your case, the questionable page).
Hi Mark,
That works well for the failed logins query on its own
GET <shard>/_search
{
"query": {
"bool": {
"filter": [
{
"bool": {
"should": [
{
"match_phrase": {
"request": "/login"
}
},
{
"match": {
"response": 401
}
}
],
"minimum_should_match": "100%"
}
}
]
}
},
"aggs": {
"significant_ips": {
"significant_terms": {
"field": "clientip.keyword"
}
}
}
}
But is there a way to combine this with a query that searches for the same IP hitting another URL besides the login?
Glad to hear it’s working for you.
The request shown above will return multiple IPs not just one.
If you are wanting to validate how many non-login accesses each ip has done the bg_count part of the response should help with that.
However if you are wanting to see what else these bad guys are attempting to access that differs from the good guys visiting your site, I demo that at 1h16m into this video. This is using the graph api in the backend to “walk out” the connections between visitors and urls. You can use that api or write custom code in your client to mimic the behaviour with multiple search calls.
Oh okay, so if I'm reading this correctly, doc_count is the number of /login hits, and bg_count is the total number of pages hit by the IP. So if I wrote a script that says (bg_count - doc_count) == 0 then the query should only return buckets with IPs that hit the /login url and nothing else, correct?
Thanks for the help!
Correct but see the caveats about the accuracy in the significant terms docs.
Also bg_count is the number of accesses (docs) not the number of unique pages accessed.
Good to know, thank you.
Is there a way that I can add a script to the aggregation to only return buckets that have a ratio of failed logins / bg >= some number?
Significant terms agg can take a scripted scoring heuristic which you could make return zero when appropriate.
Thanks for all the help. I found that the percentage {} field worked well enough for my needs.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.