How to enforce order of events in cardinality aggregation

Hello,

I have a case where I need to enable alerting when there is an event "FUN" is followed by another event "UPE" from a particular IP. First I have filtered the data for the event codes FUN and UPE , then I grouped the data by IP (terms aggregations) and then I tested the bucket count for event code (cardinality aggregation).

Here is my watcher:

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "message-logs-model-tomcat-*"
        ],
        "types": [],
        "body": {
          "_source": true,
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "request.content": "FUN UPE"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "lte": "now",
                      "gte": "now-{{ctx.metadata.window}}"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "group_by_ip": {
              "terms": {
                "field": "headers.com-transamerica-request-ip.keyword"
              },
              "aggs": {
                "group_by_code": {
                  "cardinality": {
                    "script": "def path1=params._source.request.content; int i =  path1.indexOf('code');String s = path1.substring(i, path1.length()); int p = s.indexOf(',');int l= s.indexOf(':');String dummy1 = s.substring(l+1,p-1);String dummy2 = dummy1.trim();String final = dummy2.substring(1);return final;"
                  }
                }
              }
            }
          },
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.group_by_ip.buckets": {
        "path": "group_by_code.value",
        "gt": {
          "value": 1,
          "quantifier": "some"
        }
      }
    }
  },
  "actions": {
    "email_administrator": {
      "transform": {
        "script": {
          "source": "List l = new ArrayList(); int count=1; for(item in ctx.payload.aggregations.group_by_ip.buckets) { if(item.group_by_code.value>1){l.add(item); count++;}} return ['user_info':l];",
          "lang": "painless"
        }
      },
      "email": {
        "profile": "standard",
        "from": "'test1@abc.com'",
        "priority": "high",
        "to": [
          "'test2@abc.com'"
        ],
        "subject": "Model Baseline - Encountered 'Forgot User Name' followed by 'Added Alternate Email'",
        "body": {
          "html": "<head><h4> Encountered 'Forgot User Name' followed by 'Added Alternate Email'</h4></head>{{#ctx.payload.user_info}}<table border=1 align=center><tbody><tr><th>IP Address</th><td>{{key}}</td></tr></tbody></table><p></p>{{/ctx.payload.user_info}}"
        }
      }
    }
  },
  "metadata": {
    "threshold": 30,
    "interval": "30m",
    "window": "30m"
  }
}

But the issue is, I need to alert only when "FUN" is followed by "UPE", not the other way. But this watcher alerts regardless the order of the events. Please let me know if I can enforce the order of events in cardinality aggregation or any other better ways to do this.

Thanks!
SV

Hello @saranyav

I understand that you have events labeled with either FUN or UPE which have timestamp and IP fields.
Given this, you would like to trigger the watch if, in a certain time window, there exists two events, for the same IP address, where the chronologically first event is labeled FUN and the second one is labeled UPE.

I see that you first filter events by these two labels and then do a terms aggregation on the ip field. I reckon this is all good so far. Next follows a cardinality sub aggregation which I don't dig. Instead, I would add two subsequent sub-aggregations, the first one on the labels UPN and FUN, and the innermost one a min/max aggregation on the timestamp field.

With this in place, the script condition will check, for every IP bucket, if the max value of the innermost metric aggregation for the FUN label is greater than the min value of the innermost metric aggregation for the UPN label.

Hope that helps.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.