Threshold Detection Ignoring Group By Field

Hi

I have upgraded to 7.11.0 and now 7.11.1 and I think there is an issue with Threshold detections.

I have a simple detection based on my FW logs. It looks for flow_denied or flow_dropped messages and raises a detection if a single source.ip has >= 500 events.

So the query is straight forward and the group by field is source.ip.

This used to work fine and I have a timeline template which was showing me the expected results and working well.

The rule now seems to fire for almost anything and is missing many of the fields. Many of the results that are returned don't even have a source.ip field.

The query is correct, I can apply the same filter and build a visualisation and it shows me the results as expected.

I'm no expert on this so may be wrong here...however, I have viewed my detection rule, everything is in place and I click the preview which shows it would be very noisy, as shown below. If I then click the Inspect button related to that preview is shows me the query that was sent, also shown below.

I don't fully follow the query language, however I can't see anything in there that references the grouping option of source.ip and a threshold of 500 events. At a minimum, I would expect to be able to search that block of text and find source.ip in there but it's not.

As I say, possible my misunderstanding but if this is not the issue, there does seem to have been a problem introduced with threshold detections since 7.10.2.

I have also tried creating a new rule in case the old one failed to migrate from 7.10.2 to 7.11.x for some reason but the result is the same.

Thanks in advance

Phil

{
  "aggregations": {
"eventActionGroup": {
  "terms": {
    "field": "event.category",
    "missing": "All others",
    "order": {
      "_count": "desc"
    },
    "size": 10
  },
  "aggs": {
    "events": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "112500ms",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": 1613742526822,
          "max": 1613746126822
        }
      }
    }
  }
}
  },
  "query": {
"bool": {
  "filter": [
    {
      "bool": {
        "must": [],
        "filter": [
          {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "match": {
                          "event.module": "panw"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                },
                {
                  "bool": {
                    "should": [
                      {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "event.action": "flow_denied"
                              }
                            }
                          ],
                          "minimum_should_match": 1
                        }
                      },
                      {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "event.action": "flow_dropped"
                              }
                            }
                          ],
                          "minimum_should_match": 1
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        ],
        "should": [],
        "must_not": []
      }
    },
    {
      "range": {
        "@timestamp": {
          "gte": "2021-02-19T13:48:46.822Z",
          "lte": "2021-02-19T14:48:46.822Z",
          "format": "strict_date_optional_time"
        }
      }
    }
  ]
}
  },
"size": 0,
"track_total_hits": true
}

Hi @PhilA! Thanks for the question, and sorry you're experiencing problems... I wanted to clarify a few things regarding changes in functionality in 7.11+.

We did update the functionality in 7.11 so that the fields queried in the original events will NOT be reflected in the signals. This was because the fields are not necessarily the same value across all matches, so it was ambiguous (wildcards can occur in the queries, for example)... that functionality is now provided by the timeline (when you click 'investigate in timeline', the original events are pulled back and you can see everything that matched).

Additionally, the Preview functionality has never incorporated the grouping functionality, as far as I know... it just summarizes the matching events, not the threshold signals themselves... this is a bit confusing, I admit. And feedback on that is most certainly welcome.

Does this address your concerns, or are there additional problems? You mentioned that your rule is alerting on almost anything? You should get 1 signal per bucket/group every time the rule fires over a set of events that exceeds your threshold. If you're seeing extraneous signals, I'm happy to help you investigate.

Thanks!
Madi

@PhilA I may be wrong about the preview functionality... it looks like there may be a bug in parsing the form values that's preventing the bucketing code path from being hit when 'Preview' is selected. Investigating...