Examples for building percolator with aggregations

Elastic Stack Elasticsearch
1

Hi,

I have a stream keep feeding elasticsearch for the following events:

POST /test_index/test_type/_bulk {"index":{"_id":"1"}} {"timestamp": "2015/07/22 08:00:00", "host": "www.google.com", "hits": 100} {"index":{"_id":"2"}} {"timestamp": "2015/07/22 08:01:00", "host": "www.google.com", "hits": 20} {"index":{"_id":"3"}} {"timestamp": "2015/07/22 08:02:00", "host": "www.facebook.com", "hits": 200} {"index":{"_id":"4"}} {"timestamp": "2015/07/22 08:03:00", "host": "www.yahoo.com", "hits": 80}

i.e., each document contains timestamp, host and hits.

I'm trying to figure out how to use percolator to build an alerting so that it's triggered when the following conditions are met:

  1. During the past 5 minutes, any host's average hits exceeds 100;
  2. During the past 10 minutes, the difference between the max hits and the min hits for host www.google.com exceeds 50.

This specific use case is hypothetical, but in general what I'm looking for are:

  1. How can I build a percolator with aggregations? The percolator documents indicate it's possible but I didn't find any examples.
  2. How can I pass multiple documents to a percolator? All existing examples just pass a single document to a percolator, but the alerting conditions for my case requires aggregations across multiple documents.
  3. As a variation of #2, is it possible to pass multiple existing indexed documents to a percolator?

It would be appreciated if anyone can shed some light on this.

Thanks,
Frank

1 Like
2

Anyone has any insight please? To me it is a fairly common use case, and I don't believe I'm the first person running into this.

3

Imagine a real estate agent with an index full of houses for sale ("supply") and a bunch of buyer details, registered as percolator queries looking for different sorts of houses ("demand") and with metadata describing the buyer eg date registered, Is-part-of-chain etc. this metadata can be provided as part of the json that contains a percolator query.
With this setup the estate agent can tell a prospective seller the level of demand for his house before he signs up. Given the house details, a percolate call can use aggregations to summarise the metadata on the buyers. A query on the percolate request could also rank the interested buyers based on their metadata eg how recently they registered interest.
This is demand-side aggregation in a supply/demand relationship.

What you are after sounds more like the newly launched "Watcher".

4

Hi Frank,
AFAIK, percolators have no context outside of the document you are
percolating. You might be able to do something with scripts, but I dont
think you can kick another search off a script (at least it doesnt sound
like a good idea...)