[ Threshold Rule ]: Unexpected result

[TheHunter1]

Hello,

I would like to create a Threshold rule to detect files downloaded more than 1 Go.
So in my firewall logs I have the field rcvd where I can find this value, and I to test that, I tested to download a file with 1.7 Go, and I could see that in the discover app as we can see in the picture below:

But when I try to create an threshold rule to detect that, and I run preview results, it doesn't detect it.

I have configured it like that:

index patterns : firewall-*
Custom query: rcvd: *
Field : rcvd
Threshhold : it detect something when I use 1000 but more than 4000 it not detect anaything

Could you tell me please why in the discover section I am seeing the rcvd which is equal to 1844015137 but the threshold job couldn't see it ?

Thanks for your help

[TheHunter1]

I tried to create the same rule passing by : Management → Stack Management → watcher → Create → Create threshold alert.

and now it detected it and it's workign fine as you can see here:

(I used here last 2 hours, cause it passed 1 hour since I tested it, so it's not for that, that it didn't appear in the first rule)

Could you tell me please, why the first threshold alert didn't work? is it a bug or it's just me who didn't know how to configure it !

[yctercero (Yara Tercero)]

Hi there @TheHunter1 !

To make sure I understand, your finding is:

• Threshold rule worked as expected when the threshold value was under 4000 for field rcvd
• Creating an alert via watcher you were able to get it to fire on the expected value of rcvd being >= 4000

Does that sound right? My question here would be, are you wanting to find records where the value of rcvd is greater than 4000 or are you wanting to be alerted when there are >= 4000 records that match your query within your rule timeframe?

For example, per the documentation, for threshold rules if the Field is source.ip and its Threshold is 10 , an alert is generated for every source IP address that appears in at least 10 of the rule’s search results.

Would you mind sharing the query used for discover and the query used for the preview functionality in the rule creation process? You can find the inspect button in the top right portion of the preview.

You can compare the query being used in the rule preview to the query you used in discover to ensure that they are asking the same question of your data.

[TheHunter1]

Thanks for your answer @yctercero,

In fact I want to be alerted when there is a log with rcvd value greater than 268435456, which means that a user has downloaded a file with more than 2 Go.

so for example I run the below query in the Discover app: rcvd >= 268435456 for the last 1 day logs, and I got this result:

and here is what I configured in the alerting app:

And when I clicked on preview data, I didn't go any hits

and here is the query:

{
  "aggregations": {
    "eventActionGroup": {
      "terms": {
        "field": "rcvd",
        "order": {
          "_count": "desc"
        },
        "size": 10,
        "min_doc_count": "268435456"
      },
      "aggs": {
        "events": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "2699999ms",
            "min_doc_count": 0,
            "extended_bounds": {
              "min": 1610525262930,
              "max": 1610611662929
            }
          }
        }
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "exists": {
                        "field": "rcvd"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ],
            "should": [],
            "must_not": []
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-01-13T08:07:42.930Z",
              "lte": "2021-01-14T08:07:42.929Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ]
    }
  },
  "size": 0,
  "track_total_hits": true
}

For me I thought that the Threshold Alerting, will send an alert everytime the value of rcvd be greater than 268435456, but as I see in the query generated automatically, if I understand well, the alert will be generated if "min_doc_count": "268435456" !!

[yctercero (Yara Tercero)]

That's correct! So as you can see in the query you shared from the rule creation preview, threshold is focused on doc count. The idea here being, for example, if I as a user wanted to ask "don't alert me every single time some value appears, just alert me when it occurs more than 'x' times in my rule timeframe". In other words, it is more about aggregations rather than a particular value threshold.

That said, you can accomplish the same as you did in Discover using the custom rule query.

[TheHunter1]

Thanks for your explanation, now I understand well the difference between the 2 alerts