I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).
We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.
We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.
Please let me know, if I want to give any more information.
Percolator runs incoming documents (documents getting indexed) against
stored Percolator queries as documents are being indexed (i.e. not
periodically). However, the "interval aspect" of your requirement
could be implemented in your application, which can use Percolator to
find out when documents match queries, and then buffer them in
appropriate buckets.
I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).
We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.
We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.
Please let me know, if I want to give any more information.
I've implemented a similar system in the past using custom code. We chose
to:
Generate the percolation matches as documents came in
Persist the matches consisting of something like:
document_id, percolation(subscription)_id, time_of_match,
delivery_status
3a) Deliver real-time events immediately
3b) Invoke a query for any matches for the given subscription_id from a
scheduler (quartz or whatever you prefer)
I believe I'd do the same thing with ES if I was using percolation. I've
been rethinking it as we may implement one in the coming weeks.
I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).
We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.
We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.
Please let me know, if I want to give any more information.
Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.
On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:
Hi Vibin,
I've implemented a similar system in the past using custom code. We chose to:
Generate the percolation matches as documents came in
Persist the matches consisting of something like:
document_id, percolation(subscription)_id, time_of_match, delivery_status
3a) Deliver real-time events immediately
3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)
I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.
I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).
We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.
We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.
Please let me know, if I want to give any more information.
Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.
On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:
Hi Vibin,
I've implemented a similar system in the past using custom code. We chose to:
Generate the percolation matches as documents came in
Persist the matches consisting of something like:
document_id, percolation(subscription)_id, time_of_match, delivery_status
3a) Deliver real-time events immediately
3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)
I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.
I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).
We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.
We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.
Please let me know, if I want to give any more information.
Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.
On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:
Hi Vibin,
I've implemented a similar system in the past using custom code. We chose to:
Generate the percolation matches as documents came in
Persist the matches consisting of something like:
document_id, percolation(subscription)_id, time_of_match, delivery_status
3a) Deliver real-time events immediately
3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)
I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.
I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).
We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.
We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.
Please let me know, if I want to give any more information.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.