A watch Alert on an Aggregation's Aggregation AKA an array of arrays

Elastic Stack Elasticsearch
1

I need help creating the correct compare to trigger with watch.
The bolded doc count is what I need to trigger the watch however it appears multiple times.

I am trying to create a watch on count of dst_ip by each client_ip. Thus I need the watch to compare to compare the whole array of client_ip's and then each of the sub arrays of each dst_ip count for each of the client ips.
This is the aggregation I use on firewall logs:

{
          "size": 0,
          "aggs": {
                                    "last_15m": {
                                      "range": {
                                        "field": "@timestamp",
                                        "ranges": [
                                          {
                                            "from": "now-15m",
                                            "to": "now"
                                          }
                                        ]
                                      },
                                      "aggs": {
                                        "denied": {
                                          "filter": {
                                            "term": {
                                              "action": "denied"
                                            }
                                          },
                                        "aggs": {
                                          "client_ips": {
                                            "terms": {
                                              "field": "client_ip",
                                              "size": 10
                                            },
                                            "aggs": {
                                              "dst_ip_per_client": {
                                                "terms": {
                                                  "field": "dst_ip",
                                                  "size": 10
                                                }
                                              }
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                            },
        "indices": [
          "cisco-asa*"
        ]
      }
    }
  },

The aggregation I use gets info back with the following: (I replaced the client_ip and dst_ip with \client_ip\ \dst_ip\)

{
"ctx" : {
"metadata" : null,
"watch_id" : "inlined",
"payload" : {
"_shards" : {
"total" : 146,
"failed" : 0,
"successful" : 146
},
"hits" : {
"hits" : ,
"total" : 937275493,
"max_score" : 0.0
},
"took" : 3551,
"timed_out" : false,
"aggregations" : {
"last_15m" : {
"buckets" : [
{
"from_as_string" : "2018-01-02T20:53:41.512Z",
"doc_count" : 34862,
"to_as_string" : "2018-01-02T21:08:41.512Z",
"from" : 1.514926421512E12,
"to" : 1.514927321512E12,
"denied" : {
"doc_count" : 3788,
"client_ips" : {
"doc_count_error_upper_bound" : 40,
"sum_other_doc_count" : 2078,
"buckets" : [
{
"doc_count" : 304,
"dst_ip_per_client" : {
"doc_count_error_upper_bound" : 1,
"sum_other_doc_count" : 126,
"buckets" : [
{
"doc_count" : 48,
"key": \dst_ip\
}
]
"key": \Client_ip\
}
goes on some more with more buckets for each client_ip and then an array with the top ten dst_ip's on >that ip......

Any help that could be given would be great!

2

Would using the script condition be the way to do it?
Can I do a loop and then another loop to go through each of the buckets?
like:

for each client ip {
for each dst_ip{
if the doc count > 10{
trigger;
} } }

3

hey,

you can use the min_doc_count in the terms aggregation, so you only need to check for the bucket size.

Also, you should move the time range and term filter into the query part.

The condition now basically boils down to check the size of the first bucket like this

return ctx.payload.aggregations.client_ips.buckets.size() > 0 && ctx.payload.aggregations.client_ips.buckets[0].dst_ip.buckets.size() > 0

The above one is untested, but should give you an idea.

Also, please properly format your snippets, the above is super hard to read. Alternatively just use a gist.

Thanks!

--Alex

4

Thank you! I will try this out and will attempt to reformat my snippets to be more human readable.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.