SIEM Threshold - unique values

Elastic Security SIEM
1

Hi

I am playing with the SIEM capability and have been using it since it was released with custom queries. I am now looking at Threshold based detection in v7.9 - something I think will be very useful.

I'm not sure if this is possible at the moment or a future capability but is there a method of the threshold considering aggregations and unique values?

There are a few use cases but as an example, I would like to know if a user connects inbound over the VPN from multiple locations. So if I log in from home on IP 1.2.3.4, I would not expect my account to also log in from overseas IP 5.6.7.8 within the query timeframe.

I want to query for LOGIN_SUCCESS events and the THRESHOLD would need to be PUBLIC IP >=2. There are 2 problems here, it needs to GROUP by USER and only look at UNIQUE values for the public IP. I don't want an alert if a user logs in twice from the same IP, generating multiple log events.

I basically want to be able to use the power of visualisations and effectively build a Data Table with rows for field USER and have the aggregation based on UNIQUE VALUE for public IP. Then alert when this value is >=2.

I hope that makes sense.

I have tried to build a query in Discover using DSL and aggregations but the fact that there are unique timestamps I think means I can't aggregate everything up enough. Below shows the part of DSL I was using (I'm still fairly new to this) which almost got me there but not quite. This seems to aggregate based on user and IP ok but the unique times still show in Discover.

{ "query": { "match_phrase": { "event.action": "login" } }, "aggs": { "2": { "terms": { "field": "source.user.name.keyword", "order": { "1": "desc" }, "size": 1 } }, "3": { "terms": { "field": "client.ip", "order": { "1": "desc" }, "size": 1 } }

Thanks in advance

(code above looked a little odd in the format so also shown below using blockquote as it seemed easier - fairly new to this :slight_smile:

{
"query": {
"match_phrase": {
"event.action": "login"
}
},
"aggs": {
"2": {
"terms": {
"field": "source.user.name.keyword",
"order": {
"1": "desc"
},
"size": 1
}
},
"3": {
"terms": {
"field": "client.ip",
"order": {
"1": "desc"
},
"size": 1
}
}

1 Like
2

Hi Phil, we are glad you are trying out the new Threshold detection rules in 7.9

I'm not sure if this is possible at the moment or a future capability but is there a method of the threshold considering aggregations and unique values?

This seems like a useful capability, but is not possible using the current 7.9 Threshold based rules. We'll investigate this as a possible enhancement for a future release.

In the meantime, I'll let others with more expertise in using Elasticsearch aggregations comment about your Elasticsearch DSL query. :smile:

1 Like
3

I think that you might be starting to get into a bit of an area where machine learning jobs and rules that alert you based around it might be a better way to detect this.

For example there is this one:
https://www.elastic.co/guide/en/siem/guide/current/unusual-windows-remote-user.html

Those can be adapted to your needs and tweaked at either the machine learning level or the rule level depending on what you want to change to detect the remote user if you need them but I think as-is it might work pretty well for you.

4

Hi,

Thank you for your reply.

I have had a look at the ML rules but not yet used them. I have some more data to get processing then I will activate the 30 day trial and give ML a full test. I am currently running Elastic alongside our existing SIEM tool which has been in place for several years so I want Elastic to be at its best before I give them a full side-by-side comparison.

Elastic SIEM has been a great addition to the tool and Threshold queries was missing, however as a user, I do feel it would benefit from the additional capabilities mentioned before around grouping by specific fields. This would be powerful and is a capability that exists in other SIEM tools.

I think the ML rules will be nice, however I do believe this is a statistical based query and not necessarily a ML one. If I'm honest, based on other experiences of ML/AI, I am not yet ready to leave queries up to the machine when it should be a fairly simple query to do with the data.

Thank you for your work

5

Hi Phil,

we are currently developing a rule that, in spirit, tries to do the same thing yours does.
Our use-case is a bit different, we want to compare the source and destination field to each other and ignore events where the two fields are the same but I think the ideas we had might be of help to you.

Currently we are trying and evaluating three different approaches:

Logstash JDBC Lookups
The JDBC Filter from Logstash allows you to make lookups either against a local database or a remote one (static vs streaming plugin) and enrich an event with certain information. If you are able to create a database with your logins you could potentially query that and add a field to every event where the IP addresses are different to the Database entry.

Make use of the List API
Another way we're trying to make this work is by looking if we can somehow populate a list through the lists API with the first IP address a user uses to authenticate to us. Every other remote connection from that user with an IP address that is different from that list entry should throw an alert.

Make use of a painless script
Since what we are comparing the values of two fields in an event to each other for our use-case we thought about making use of the score query provided by the painless scripts.
This would allow us to compare the values of two fields to each other and if they differ, we get an alert.

The main issue I see with your use-case is that you'd have to hold onto that piece of information for quite a long time. For the entirety of the day at least actually.
I have yet to find a way that does not feel hacky to me in the ES SIEM to persistently hold on to data like that. However I do hope that some of our approaches might give you an idea on how to tackle your issue.

Cheers,
madduck

6

Thank you for the reply.

I quite like the list idea but think a useful addition to that would be to allow for a expiration period on the list, for example the scenario I gave may have an 8 hour expiration. So if a user successfully authenticates, their username and IP are added to the list and set to expire 8hrs later. If subsequent authentication messages are sent, that expiration is extended for 8hrs from the new event. If that same user authenticates from a different IP while their list entry is active, an event is generated.

As a user I would need to be able to design the policies of the list and what field(s) are required for it. I know there is some list stuff in the UI in v7.9 but I haven't had a good look at it yet but a quick look suggested it could support a single field type where the scenario I gave would need to be able to track user and IP.

Appreciate your response, I know my focus was around 1 scenario but I can see many others where it would work and offer value.

Thanks

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.