How to monitor cumulative totals and send distinct alerts

[Chris_Blackwell]

We're using elastic to analyze some voip call traffic and I'm trying to figure out how to use watcher to monitor some limits we'd like to impose.

So an example would be;

Send me an alert when a calling number makes more than X minutes of calls in a 24hr period. Each time a different calling number breaks that threshold, send me a new alert.

I can query for numbers that are breaking the limit using a combination of terms, sum and bucket_selector aggregation, and then transform that into a series of new documents which I can index.

But what I would like is to execute a webhook action for each of those new documents, and have the throttle period apply per doc. Is it possible to do this all in watcher?

[Christian_Dahlqvist]

If you write a document into an index for every number you alert on, you should be able to use a chained input to remove any users that have already been alerted on within the throttle period. There is simple example in GitHub that show an example of how to remove results from an initial result set using a chained input.

[Chris_Blackwell]

Thanks @Christian_Dahlqvist.

I'm storing new hits in an index and successfully using chained inputs to get the disjoint of new and old hits. This is working ok atm but I'm wondering how well it will scale if I have must_not.terms clause with hundreds or potentially thousands of entries

Also, is it possible to execute a webhook action for each document, like it is with the index action?

[spinscale (Alexander Reelsen)]

Hey,

the webhook action only supports single HTTP requests. Maybe it would help if you send the watch results to logstash using the webook action (and an optional transform) and process it step by step there via a custom filter and the http input.

--Alex

[Chris_Blackwell]

Ok, after half a day of fiddling, I managed to get logstash to do what I needed.

in case it's of use to anyone, here's a basic config that takes an array of objects and converts it to seperate http calls

	input {
		http {
			codec => json
			port => 8989
			type => "http_fanout"
		}
	}
	output {
		if [type] == "http_fanout" {
			http {
				url => "%{url}"
				http_method => "post"
				format => "message"
				message => "%{body}"
			}
		}
	}

takes an input like

[
	{
		"url": "http://requestb.in/14dasrg1", 
		"body": {"name": "alice"}
	},
	{
		"url": "http://requestb.in/14dasrg1", 
		"body": {"name": "bob"}
	}
]

I found that http_method will only take a static string, so you can't pass the method to be used in and use a variable like %{method}, so if you wanted to support other methods you'd have to include a if/else block around the http {} output