# Threshold detection not working with group by

**URL:** <https://discuss.elastic.co/t/threshold-detection-not-working-with-group-by/274362>\
**Category:** SIEM\
**Tags:** elastic-stack-alerting\
**Created:** [May 28, 2021, 4:57pm UTC](https://discuss.elastic.co/t/threshold-detection-not-working-with-group-by/274362 "2021-05-28T16:57:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Billz1026](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@Billz1026](https://discuss.elastic.co/u/Billz1026)\
**Post date:** [May 28, 2021, 4:57pm UTC](https://discuss.elastic.co/t/threshold-detection-not-working-with-group-by/274362/1 "2021-05-28T16:57:58Z")

</div>

Hi All,

I am using Elastic version 7.12. I want to create a detection rule based on the threshold. My requirements is as follows.

> Identify possible Bruteforce attack coming from same source IP. If there are more than 10 incidents of 4625 event log from same source IP, an alert should trigger.

For that I have configured a detection rule as below.

 ![Elastic threshold rule issue](https://us1.discourse-cdn.com/elastic/original/3X/7/4/74bf5574a200934a9b4137770c538fc82dd5e9ff.png)

But when I run the preview results, it is giving zero. And when I inspect the response I see below.

```
{
"took": 8,
"timed_out": false,
"_shards": {
"total": 39,
"successful": 38,
"skipped": 38,
"failed": 1,
"failures": [
  {
    "shard": 0,
    "index": "winlogbeat-7.12.0-2021.05.28-000037",
    "node": "a5GvVwfnSL6vIY9IjCRsqw",
    "reason": {
      "type": "illegal_argument_exception",
      "reason": "'All others' is not an IP string literal."
    }
  }
]
},
"hits": {
"total": 0,
"max_score": 0,
"hits": []
 }
}

```

The query is as below.

```
      {
  "aggregations": {
    "eventActionGroup": {
      "terms": {
        "order": {
          "_count": "desc"
        },
        "size": 10,
        "field": "source.ip",
        "missing": "All others"
      },
      "aggs": {
        "events": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "112500ms",
            "min_doc_count": 10,
            "extended_bounds": {
              "min": 1622216794290,
              "max": 1622220394290
            }
          }
        }
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "match": {
                        "event.code": "4625"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ],
            "should": [],
            "must_not": []
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-05-28T15:46:34.290Z",
              "lte": "2021-05-28T16:46:34.290Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ]
    }
  },
  "size": 0
}

```

Could someone please help me to solve this issue.

Thanks.  
Someunguy1026

---

<div class="post-metadata">

**Author:** ![spong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spong/32/54343_2.png) [@spong](https://discuss.elastic.co/u/spong)\
**Post date:** [May 28, 2021, 7:59pm UTC](https://discuss.elastic.co/t/threshold-detection-not-working-with-group-by/274362/2 "2021-05-28T19:59:11Z")

</div>

Hey there @Billz1026 👋

So this is a known issue with Threshold Rules and the Preview functionality. Your detection rule should still function without issue though, so you can try creating your rule and observing the alerts generated. Please see the below two issues for all the details around this. 🙂

> <https://github.com/elastic/kibana/issues/100433>
>
> \*\*Describe the bug:\*\*
> Users are unable to view preview results for Threshold ru…les configured with a \`group by\` field of type \`ip\`, such as \`source.ip\`.
> 
> \*\*Kibana/Elasticsearch Stack version:\*\*
> 7.12, 7.13
> 
> \*\*Steps to reproduce:\*\*
> To reproduce:
> 1. Create a test index with an \`ip\` field like this:
> \<details\>
> \<summary\>Dev Console code\</summary\>
>       
> \`\`\`
> PUT test-index-1
> {
> "mappings": {
> "properties": {
> "source": {
> "properties": {
> "ip": {
> "type": "ip"
> }
> }
> }
> }
> }
> \`\`\`
> \</details\>
> 
> 2. Index a few documents like this:
> \<details\>
> \<summary\>Dev Console code\</summary\>
>       
> \`\`\`
> POST test-index-1/\_doc
> {
> "@timestamp": "2021-05-20T18:10:49.337Z",
> "source": {
> "ip": "1.1.1.1"
> }
> }
> \`\`\`
> \</details\>
> 
> 3. Begin to create a threshold rule and configure it like this, with \`source.ip\` as the \`group by\` field:
>        
> !\[image\](https://user-images.githubusercontent.com/616158/119083930-c299a380-b9b5-11eb-9b1b-1b8805a25e1f.png)
> 
> 
> 4. Click the \`Preview Results\` button.
> 
> \*\*Current behavior:\*\*
> The following error gets displayed
> \`\`\`
> "failures": \[
> {
> "shard": 0,
> "index": "filebeat-8.0.0-2021.04.23-000010",
> "node": "UmFIfD\_iQfepBl6sUyEk2g",
> "reason": {
> "type": "illegal\_argument\_exception",
> "reason": "'All others' is not an IP string literal."
> }
> }
> \]
> \`\`\`
> 
> \*\*Expected behavior:\*\*
> No error should be displayed and user should be able to see Preview results.
> 
> \*\*Screenshots (if relevant):\*\*
> \*\*7.12.0\*\*
> !\[image\](https://user-images.githubusercontent.com/616158/119082955-efe55200-b9b3-11eb-99f9-c6bff02419ae.png)
> 
> \*\*master\*\*
> !\[image\](https://user-images.githubusercontent.com/616158/119082895-da702800-b9b3-11eb-855a-39fd015c86e5.png)

> <https://github.com/elastic/kibana/issues/98350>
>
> \*\*Describe the bug:\*\*
> In our security solutions application we use this type of… query in a lot of places and mark the UI as being "others" when in reality this is not "others". It is actually the missing records and values:
> \<img width="812" alt="Screen Shot 2021-04-26 at 11 00 06 AM" src="https://user-images.githubusercontent.com/1151048/116122124-9b81d780-a67e-11eb-896a-91d78c73ab9d.png"\>
> 
> \`\`\`ts
> "alertsGroup": {
> "terms": {
> "field": "event.module",
> "missing": "All others",
> "order": {
> "\_count": "desc"
> },
> "size": 10
> },
> \`\`\`
> 
> Top N record queries:
> \<img width="825" alt="Screen Shot 2021-04-26 at 11 02 03 AM" src="https://user-images.githubusercontent.com/1151048/116122511-1cd96a00-a67f-11eb-9194-160b3f91f9f5.png"\>
> 
> \`\`\`ts
> "terms": {
> "field": "destination.ip",
> "missing": "0.0.0.0",
> "order": {
> "\_count": "desc"
> },
> "size": 10
> },
> \`\`\`
> 
> We also using missing inside of alerting and a few other areas. However, we are not consistent with misusing missing records as an "others". Not all our widgets use this paradigm. Some widgets use it, some do not use it.
> 
> Another bigger issue is that this UI/UX is telling everyone that this is "others" when in reality is showing "missing" records. To do true others, we have to use two queries like Lens has. See this:
> https://github.com/elastic/elasticsearch/issues/12411#issuecomment-508502044
>  
> Screen shot of lens doing two queries to get true others:
> 
> \<img width="870" alt="Screen Shot 2021-04-13 at 3 44 50 PM" src="https://user-images.githubusercontent.com/1151048/116123016-b43ebd00-a67f-11eb-80a6-e2272298c0cb.png"\>
> 
> We should change our queries to utilize a true "others" and remove the "missing" based queries.
> 
> \*\*Kibana version:\*\*
> 7.6+
> 
> \*\*Elasticsearch version:\*\*
> 7.6+

Cheers!  
Garrett

---

<div class="post-metadata">

**Author:** ![Billz1026](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@Billz1026](https://discuss.elastic.co/u/Billz1026)\
**Post date:** [May 31, 2021, 3:57am UTC](https://discuss.elastic.co/t/threshold-detection-not-working-with-group-by/274362/3 "2021-05-31T03:57:27Z")

</div>

Hi Garrett,

Thanks for the feedback.

BR,  
Someunguy1026

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2021, 3:58am UTC](https://discuss.elastic.co/t/threshold-detection-not-working-with-group-by/274362/4 "2021-06-28T03:58:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.