Percolator for alerts between specific time intervals?


(vibin) #1

Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


(Otis Gospodnetić) #2

Hi,

Percolator runs incoming documents (documents getting indexed) against
stored Percolator queries as documents are being indexed (i.e. not
periodically). However, the "interval aspect" of your requirement
could be implemented in your application, which can use Percolator to
find out when documents match queries, and then buffer them in
appropriate buckets.

Otis

On Jan 26, 10:37 pm, vibin dhas vibind...@gmail.com wrote:

Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


(Michael Sick) #3

Hi Vibin,

I've implemented a similar system in the past using custom code. We chose
to:

  1. Generate the percolation matches as documents came in
  2. Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match,
    delivery_status
    3a) Deliver real-time events immediately
    3b) Invoke a query for any matches for the given subscription_id from a
    scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've
been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas vibindhas@gmail.com wrote:

Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


(Shay Banon) #4

Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.

On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:

Hi Vibin,

I've implemented a similar system in the past using custom code. We chose to:

  1. Generate the percolation matches as documents came in
  2. Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match, delivery_status
    3a) Deliver real-time events immediately
    3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <vibindhas@gmail.com (mailto:vibindhas@gmail.com)> wrote:

Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


(vibin) #5

Great got it...Still I have a doubt

Does Polling for changes in the indices and Hitting the indices with
search(with filter) at specific time intervals, will have same
performance?

On Jan 28, 1:53 am, Shay Banon kim...@gmail.com wrote:

Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.

On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:

Hi Vibin,

I've implemented a similar system in the past using custom code. We chose to:

  1. Generate the percolation matches as documents came in
  2. Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match, delivery_status
    3a) Deliver real-time events immediately
    3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <vibind...@gmail.com (mailto:vibind...@gmail.com)> wrote:

Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


(Shay Banon) #6

Searching (with a filter) is what I meant by polling for changes.

On Tuesday, January 31, 2012 at 6:27 AM, vibin dhas wrote:

Great got it...Still I have a doubt

Does Polling for changes in the indices and Hitting the indices with
search(with filter) at specific time intervals, will have same
performance?

On Jan 28, 1:53 am, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.

On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:

Hi Vibin,

I've implemented a similar system in the past using custom code. We chose to:

  1. Generate the percolation matches as documents came in
  2. Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match, delivery_status
    3a) Deliver real-time events immediately
    3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <vibind...@gmail.com (mailto:vibind...@gmail.com (http://gmail.com))> wrote:

Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


(system) #7