So to be specific with what we are trying to accomplish, we are using Elastic Security's detections and have a few rules set up. We then have a logstash pipeline that queries the index our detections go into every minute and then any results get taken and used to create a ticket in our ticketing system.
The issue we are running into is that for event correlation rules we are getting a new doc in the index the detection posts to for every event that correlates into the detection. So we are having a ticket get created for every individual event that is correlated, when we only want 1 ticket to be created. There is a field for each of these individual events called signal.depth that indicates whether or not it is the parent event essentially, so if there are 6 events that get correlated together the parent event will have the highest signal.depth. Additionally, all the events that are correlated together share a field signal.group.id.
Our thought was that if we could write an elasticsearch query (which is used as the input for our logstash pipeline) that only gets the event with highest signal.depth, grouped by signal.group.id then that would avoid the issue of duplicate tickets being created. The other issue however, is that for our other detections that don't come from numerous correlated events, the field signal.group.id does not exist. I've been able to put together that I believe just gives us the parent event of these correlated events, however since we are grouping by signal.group.id we are now losing all the other events that don't contain that field.
I am pretty new to elasticsearch queries, so any guidance on this would be greatly appreciated. For reference, here is the query that I currently have:
{
"size":0,
"query":{
"bool":{
"filter":[
{
"range":{
"@timestamp":{
"gte":"now-1m",
"lte":"now"
}
}
}
]
}
},
"aggs" : {
"groupByGroupID" : {
"terms" : {
"field" : "signal.group.id"
},
"aggs": {
"maxDepth": {
"max": {
"field": "signal.depth"
}
},
"aggs": {
"top_hits": {
"sort": [
{
"signal.depth": {
"order": "desc"
}
}
],
"size": 100
}
}
}
}
}
}