Generate alarm only when a condition is applied

Dear,

I'm trying to config an alarm to be sent only when a value from aggregation has a desired or higher value.

e.g. i receive this email, but it must contains only United States and France:
workstation 1 | username 1

  • United States (300)
  • France (200)
  • Irak (99)
  • China (20)

My watcher script is:

POST _xpack/watcher/watch/index1-NetbiosConnections/_execute

PUT _xpack/watcher/watch/index1-NetbiosConnections
{
	"trigger": {
		"schedule": {
      "daily" : {
        "at" : {
          "hour" : 10,
          "minute" : 0
        }
      }
		}
	},
	"input": {
		"search": {
			"request": {
				"indices": [
					"winevt-*"
				],
				"body": {
					"size": 0,
					"query": {
						"bool": {
							"filter": {
								"range": {
									"@timestamp": {
										"gte": "now-1d"
									}
								}
							}, 
							"should": [
								{
									"match_phrase": {
										"DestinationPortName": "netbios-ns" 
									}
								}
							]
						}
					},
					"aggs": {
						"group_by_workstation": {
							"terms": {
								"field": "workstation_name"
							},
							"aggs": {
								"group_by_enduser": {
									"terms": {
										"field": "enduser.keyword"
									}
								},
								"group_by_countryname": {
									"terms": {
										"field": "country_name"
									}
								}
							}
						}
					}
				}
			}
		}
	},
	"condition": {    
		"array_compare" : { 
		  "ctx.payload.aggregations.group_by_countryname.buckets" : {
		    "path": "doc_count",
		    "gte": {
		      "value": 100,
		      "quantifier": "all"
		    }
		  }
		}
	},
	"actions": {
		"email_me": {
			"throttle_period": "1d",
			"email": {
				"from": "***e-mail address***",
				"to": "***e-mail address***",
				"subject": "NetBios Connections",
				"body": {
					"html": "Total events: {{ctx.payload.hits.total}}<br><br><br>{{#ctx.payload.aggregations.group_by_workstation.buckets}} {{key}} | {{#group_by_enduser.buckets}}{{key}}{{/group_by_enduser.buckets}}<br> <ul> {{#group_by_countryname.buckets}}<li>{{key}} ({{doc_count}}){{/group_by_countryname.buckets}}</li></ul>{{/ctx.payload.aggregations.group_by_workstation.buckets}}"
				}
			}
		},
		"log_error": {
			"logging": {
				"text": "Were found {{ctx.payload.hits.total}} events"
			}
		}
	}
}

Thanks!

Hey,

first, please take your time and properly format your message. This is insanely hard to read. As you can use markdown, this should not be too hard.

The easiest solution to your problem is to specify min_doc_count in your terms aggregation. See the documentation

You could also have a script transform in your action, where you exclude all the other buckets, but I do not think that this is needed in your case.

--Alex

I've added min_doc_count, but I receive the alert even if I don't have events and e-mail is empty. Could you help me with a script example to exclude this situation?

					"aggs": {
						"group_by_workstation": {
							"terms": {
								"field": "workstation_name",
								"min_doc_count": 200
							},
							"aggs": {
								"group_by_enduser": {
									"terms": {
										"field": "enduser.keyword",
										"min_doc_count": 200
									}
								},
								"group_by_countryname": {
									"terms": {
										"field":DestinationIp_geoip. "country_name",
										"min_doc_count": 200
									}
								}
							}
						}
					}

Hey,

you can try to check for the bucket size like this using a script condition

return ctx.payload.aggregations.group_by_countryname.buckets.size() > 0

If that does not work, can you paste the output from the Execute Watch API?

--Alex

I got an error, I will split in three posts:

  {
  "_id": "index1-NetbiosConnections_5021c1a7-2ee5-49fb-8a6b-7f0984d2acf4-2017-06-08T12:00:43.500Z",
  "watch_record": {
"watch_id": "index1-NetbiosConnections",
"state": "failed",
"trigger_event": {
  "type": "manual",
  "triggered_time": "2017-06-08T12:00:43.500Z",
  "manual": {
    "schedule": {
      "scheduled_time": "2017-06-08T12:00:43.500Z"
    }
  }
},
"input": {
  "search": {
    "request": {
      "search_type": "query_then_fetch",
      "indices": [
        "index1-*"
      ],
      "types": [],
      "body": {
        "size": 0,
        "query": {
          "bool": {
            "filter": {
              "range": {
                "@timestamp": {
                  "gte": "now-1d"
                }
              }
            },
            "should": [
              {
                "match_phrase": {
                  "DestinationPortName": "netbios-ns"
                }
              }
            ]
          }
        },
        "aggs": {
          "group_by_workstation": {
            "terms": {
              "field": "workstation_name",
              "min_doc_count": 200
            },
            "aggs": {
              "group_by_enduser": {
                "terms": {
                  "field": "enduser.keyword",
                  "min_doc_count": 200
                },
                "aggs": {
                  "group_by_countryname": {
                    "terms": {
                      "field": "country_name",
                      "min_doc_count": 200
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
},
"condition": {
  "script": {
    "inline": "return ctx.payload.aggregations.group_by_countryname.buckets.size() > 0",
    "lang": "painless"
  }
},
"result": {
  "execution_time": "2017-06-08T12:00:43.500Z",
  "execution_duration": 3666,
  "input": {
    "type": "search",
    "status": "success",
    "payload": {
      "_shards": {
        "total": 30,
        "failed": 0,
        "successful": 30
      },
      "hits": {
        "hits": [],
        "total": 11266668,
        "max_score": 0
      },
      "took": 3656,
      "timed_out": false,
      "aggregations": {
        "group_by_workstation": {
          "doc_count_error_upper_bound": 84094,
          "sum_other_doc_count": 8945865,
          "buckets": [
            {
              "doc_count": 573339,
              "key": "Workstation1",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": [
                        {
                          "doc_count": 201,
                          "key": "Ireland"
                        }
                      ]
                    },
                    "doc_count": 573339,
                    "key": "Username1"
                  }
                ]
              }
            },
            {
              "doc_count": 303487,
              "key": "Workstation2",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": []
                    },
                    "doc_count": 303487,
                    "key": "Username2"
                  }
                ]
              }
            },
            {
              "doc_count": 279145,
              "key": "Workstation3",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": [
                        {
                          "doc_count": 1039,
                          "key": "Italy"
                        }
                      ]
                    },
                    "doc_count": 279145,
                    "key": "Username3"
                  }
                ]
              }
            },
            {
              "doc_count": 208481,
              "key": "Workstation4",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": []
                    },
                    "doc_count": 208481,
                    "key": "Username4"
                  }
                ]
              }
            },
            {
              "doc_count": 198518,
              "key": "Workstation5",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": [
                        {
                          "doc_count": 194055,
                          "key": "United States"
                        }
                      ]
                    },
                    "doc_count": 198518,
                    "key": "Username5"
                  }
                ]
              }
            },
            {
              "doc_count": 165420,
              "key": "Workstation6",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": []
                    },
                    "doc_count": 165420,
                    "key": "Username6"
                  }
                ]
              }
            },
            {
              "doc_count": 155430,
              "key": "Workstation7",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": []
                    },
                    "doc_count": 155430,
                    "key": "Username7"
                  }
                ]
              }
            },
            {
              "doc_count": 147325,
              "key": "Workstation8",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": []
                    },
                    "doc_count": 147325,
                    "key": "Username8"
                  }
                ]
              }
            },
            {
              "doc_count": 147083,
              "key": "Workstation9",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": []
              }
            },
            {
              "doc_count": 142575,
              "key": "Workstation10",
              "group_by_enduser": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "group_by_countryname": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": []
                    },
                    "doc_count": 142575,
                    "key": "Username10"
                  }
                ]
              }
            }
          ]
        }
      }
    },
   "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "index1-*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "now-1d"
                  }
                }
              },
              "should": [
                {
                  "match_phrase": {
                    "DestinationPortName": "netbios-ns"
                  }
                }
              ]
            }
          },
          "aggs": {
            "group_by_workstation": {
              "terms": {
                "field": "workstation_name",
                "min_doc_count": 200
              },
              "aggs": {
                "group_by_enduser": {
                  "terms": {
                    "field": "enduser.keyword",
                    "min_doc_count": 200
                  },
                  "aggs": {
                    "group_by_countryname": {
                      "terms": {
                        "field": "country_name",
                        "min_doc_count": 200
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "actions": []
},
"exception": {
  "type": "script_exception",
  "reason": "runtime error",
  "caused_by": {
    "type": "null_pointer_exception",
    "reason": null,
    "stack_trace": """
java.lang.NullPointerException
	at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:200)
	at org.elasticsearch.painless.Executable$Script.execute(return ctx.payload.aggregations.group_by_countryname.buckets.size() > 0 @ <inline script>:53)
	at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:123)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:95)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:85)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:390)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:274)
	at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136)
	at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:63)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"""
  },
  "script_stack": [
    "return ctx.payload.aggregations.group_by_countryname.buckets.size() > 0",
    "                                                    ^---- HERE"
  ],
  "script": "return ctx.payload.aggregations.group_by_countryname.buckets.size() > 0",
  "lang": "painless",
  "stack_trace": """
ScriptException[runtime error]; nested: NullPointerException;
	at org.elasticsearch.painless.ScriptImpl.convertToScriptException(ScriptImpl.java:181)
	at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:128)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:95)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:85)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:390)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:274)
	at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136)
	at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:63)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:200)
	at org.elasticsearch.painless.Executable$Script.execute(return ctx.payload.aggregations.group_by_countryname.buckets.size() > 0 @ <inline script>:53)
	at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:123)
	... 13 more

"""
}
  }
}

your aggregation has a different nesting structure than in my example. You have another aggregation around that, which contains the group_by_enduser field.

If you want to check for several buckets being non-null, you have to adapt the script to check for each of those buckets.

Is that what you want or did I misunderstood your aggregation?

To obtain the final result as below:

Wokstation | Username

  • Country1 (no. of events)
  • Country2 (no. of events)

i used this type of aggregation. Yes, you have understood what I want.
I want to generate an alarm only when a condition for my subbucket is met, also to keep only the buckets who are aggregated with this.

For example if doc_count for "group_by_countryname" aggregation is equal or higher than 200 only then to generate an alarm who contains the others aggregated fields.

Could you help me please with an script example?

Hey,

you need to use the allMatch call of painless (which is basically plain java), that will check for the number of documents in each sub bucket.

The examples repo has a couple of examples, like this script.

Also, you might want to read this blog post for more efficient testing and debugging of watches to speed up your testing cycle. For further debugging, running the execute watch API, with an alternative input might be very helpful, as you can share a watch without sharing or preparing any data. In order to ignore the length issues here, you can also just use a gist.

Hope this helps!

--Alex

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.