Detection threshold rule problem

[Jorge7 (Jorge)]

Hi everyone,

I'm try to create a personalize rule with the threshold parameter detection. I am using a handmade index with personal fields. This is the mapping:

{
  "login-000001" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "source" : {
          "properties" : {
            "@timestamp" : {
              "type" : "date"
            },
            "dispositivo" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "event" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "geo" : {
              "properties" : {
                "ip" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "location" : {
                  "type" : "geo_point"
                },
                "name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "proc" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "status" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "substatus" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timestamp" : {
              "type" : "date"
            },
            "usuario" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}

I am trying to detect the fail events from each user using the field "source.usuario.keyword" with threshol > 2. But in the result field signal.threshold.result.value appears empty, I only collect the count of the signal, but not the user who made more than 2 fail events.

Any help please? Best regards.

[yctercero (Yara Tercero)]

Hey @Jorge7 !

Thanks for your post. So I'm pulling from a recent related question regarding threshold rules where my teammate @madi noted the following about threshold rule alerts:

We did update the functionality in 7.11 so that the fields queried in the original events will NOT be reflected in the signals. This was because the fields are not necessarily the same value across all matches, so it was ambiguous (wildcards can occur in the queries, for example)... that functionality is now provided by the timeline (when you click 'investigate in timeline', the original events are pulled back and you can see everything that matched) [...]
The Timeline functionality for threshold rules is a little unreliable currently, but will be tightened up in the upcoming 7.12 release. You should be able to visualize all the events that made up the signal in Timeline out of the box [...]

(Threshold Detection Ignoring Group By Field)

Essentially, you should be able to view each individual event relating to that threshold alert when you pull it into your Timeline. Let us know if that helps!

Best,
Yara

[yctercero (Yara Tercero)]

I also should note that the Elastic SIEM/Security app requires your data to be indexed in an ECS-compliant format. ECS is an open source schema that specifies field names and ES data types for each field, and provides descriptions and example usage.

You'd mentioned that you have some custom fields, which works, but any that overlap with ECS and whose type does not match ECS will result in rule run time errors. You may have already been aware, but just thought I'd throw that in!

[Jorge7 (Jorge)]

Thank you for the reply Yara.

So, maybe if I downgrade to 7.10 or lower, I can obtain these values?

In case I decide to mantein the 7.11 version, how I can collect these data from Timeline? Is there an index in this regard like .siem-signals-default?

Thanks and regards!

[yctercero (Yara Tercero)]

Hi @Jorge7 -

I wouldn't say you need to downgrade. There are a lot of awesome updates and features that have been added. You can still accomplish what you're looking for using Timeline. Here's some docs on Timeline that could be helpful.

Let's say I create a threshold rule with a simple query of host.name: * and I want an alert created every time within my rule's 1 minute interval where host.name appears in 20 events or more. I now see that I'm getting some alerts. I want to see what events triggered this first alert, so I click on "Investigate in timeline":

Now Timeline pops up with everything I need populated. I see my main query, the host.name that triggered and the related events:

I'll also note that if you're interested in detecting a series of events occurring, EQL rules are awesome for that. And here's how you can investigate EQL rule events in Timeline.

[Jorge7 (Jorge)]

Thanks for the replay Yara, very interesting info.

Our problem is the group by in the detection. We want to show the users who login fail more than 5 times (for example). We establish the detection query to the login event and the user field to threshold > 5.

In the results, we can show the events with different values, but we cannot see the user who exceedes the threshold. It only appears the result value of the threshold 5, 6 or 10 but not the user.

Thanks and regards!