# Notifications from Elasticsearch when documents are added **URL:** \ **Category:** Elasticsearch\ **Created:** [August 10, 2011, 6:16am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106 "2011-08-10T06:16:05Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 6:16am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/1 "2011-08-10T06:16:05Z")
In couchDB there is a service where any external application interested in a particular document can hook to coucDB and if the sth document is available , couchDB notifies the external application. Is there any similar feature in elasticSearch ? Basically i have another service running somewhere and it is interested in documents added to elasticSearch. Thanks Vineeth ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 10, 2011, 6:23am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/2 "2011-08-10T06:23:02Z")
+1 for this feature if it doesn't exist yet ! David 😉 Le 10 août 2011 à 08:16, Vineeth Mohan [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) a écrit : > In couchDB there is a service where any external application > interested in a particular document can hook to coucDB and if the sth > document is available , couchDB notifies the external application. > > Is there any similar feature in elasticSearch ? > Basically i have another service running somewhere and it is > interested in documents added to elasticSearch. > > Thanks > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 10, 2011, 10:21am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/3 "2011-08-10T10:21:41Z")
There isn't an option for that. How do you envision the notification, as a pull or push service? With couchdb, do you refer to the \_changes API? On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com)wrote: > In couchDB there is a service where any external application > interested in a particular document can hook to coucDB and if the sth > document is available , couchDB notifies the external application. > > Is there any similar feature in elasticSearch ? > Basically i have another service running somewhere and it is > interested in documents added to elasticSearch. > > Thanks > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 10:28am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/4 "2011-08-10T10:28:02Z")
Yes , i mean the \_change API. ([Change Notifications](http://guide.couchdb.org/draft/notifications.html) - Continious notification) I am sure this feature will help many ppl in their integration process. Thanks Vineeth On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > There isn't an option for that. How do you envision the notification, as a > pull or push service? With couchdb, do you refer to the \_changes API? > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com)wrote: > > > In couchDB there is a service where any external application > > interested in a particular document can hook to coucDB and if the sth > > document is available , couchDB notifies the external application. > > > > Is there any similar feature in elasticSearch ? > > Basically i have another service running somewhere and it is > > interested in documents added to elasticSearch. > > > > Thanks > > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 10, 2011, 10:29am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/5 "2011-08-10T10:29:27Z")
This feature is tricky to implement because elasticsearch is built quite differently than couchdb, also, couchdb has it easy since its a single node. On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Yes , i mean the \_change API. > ([Change Notifications](http://guide.couchdb.org/draft/notifications.html) - Continious > notification) > > I am sure this feature will help many ppl in their integration process. > > Thanks > Vineeth > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > There isn't an option for that. How do you envision the notification, as a > > pull or push service? With couchdb, do you refer to the \_changes API? > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan \<[VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) > > > > > wrote: > > > > > In couchDB there is a service where any external application > > > interested in a particular document can hook to coucDB and if the sth > > > document is available , couchDB notifies the external application. > > > > > > Is there any similar feature in elasticSearch ? > > > Basically i have another service running somewhere and it is > > > interested in documents added to elasticSearch. > > > > > > Thanks > > > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 10:32am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/6 "2011-08-10T10:32:21Z")
If someone can give tips and clue on how o implement this , i am willing to make a open source plugin out of this. Thanks Vineeth On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > This feature is tricky to implement because elasticsearch is built quite > differently than couchdb, also, couchdb has it easy since its a single node. > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Yes , i mean the \_change API. > > ([Change Notifications](http://guide.couchdb.org/draft/notifications.html) - Continious > > notification) > > > > I am sure this feature will help many ppl in their integration process. > > > > Thanks > > Vineeth > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > There isn't an option for that. How do you envision the notification, as > > > a pull or push service? With couchdb, do you refer to the \_changes API? > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan \< > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com)\> wrote: > > > > > > > In couchDB there is a service where any external application > > > > interested in a particular document can hook to coucDB and if the sth > > > > document is available , couchDB notifies the external application. > > > > > > > > Is there any similar feature in elasticSearch ? > > > > Basically i have another service running somewhere and it is > > > > interested in documents added to elasticSearch. > > > > > > > > Thanks > > > > Vineeth ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [August 10, 2011, 2:02pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/7 "2011-08-10T14:02:40Z")
Hi Vineeth, Would it be better to do this feature outside of Elasticsearch ? I guess you already have a service which keeps adding docs to Elasticsearch. You can add a hook here to notify your other systems. I think this is better because in most cases you will not be using ES as a main data store. Regards, Mahendra On Wed, Aug 10, 2011 at 11:46 AM, Vineeth Mohan [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > In couchDB there is a service where any external application > interested in a particular document can hook to coucDB and if the sth > document is available , couchDB notifies the external application. > > Is there any similar feature in elasticSearch ? > Basically i have another service running somewhere and it is > interested in documents added to elasticSearch. > > Thanks > Vineeth -- Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [August 10, 2011, 2:04pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/8 "2011-08-10T14:04:04Z")
Hi Shay, CouchDB has distributed node support via BigCouch. They also have a \_changes interface which can be used for getting notifications on an update to any node. Regards, Mahendra On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > This feature is tricky to implement because elasticsearch is built quite > differently than couchdb, also, couchdb has it easy since its a single node. > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) > wrote: > > > Yes , i mean the \_change API. > > ([http://guide.couchdb.org/draft/notifications.html](http://guide.couchdb.org/draft/notifications.html) - Continious > > notification) > > > > I am sure this feature will help many ppl in their integration process. > > > > Thanks > > Vineeth > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > There isn't an option for that. How do you envision the notification, as > > > a pull or push service? With couchdb, do you refer to the \_changes API? > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > > > > > > > In couchDB there is a service where any external application > > > > interested in a particular document can hook to coucDB and if the sth > > > > document is available , couchDB notifies the external application. > > > > > > > > Is there any similar feature in elasticSearch ? > > > > Basically i have another service running somewhere and it is > > > > interested in documents added to elasticSearch. > > > > > > > > Thanks > > > > Vineeth -- Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 2:11pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/9 "2011-08-10T14:11:29Z")
Hello Mahendra , Thanks for your reply. I am planning to use elasticSeach as the main data source. I initially tot of using couchDB with ES. But then i found that using map-attachment plugin , i can index PDF file (and most of the widely used formats). On configuring ES to persist the attachment field , i am able to store and retrieve the documents i need in ES itself. I don't see why i should use couchDB+ES when i can store , index and retrieve document in ES itself. Thanks Vineeth On Aug 10, 7:04 pm, Mahendra M [mahendr...@gmail.com](mailto:mahendr...@gmail.com) wrote: > Hi Shay, > > CouchDB has distributed node support via BigCouch. They also have a > \_changes interface which can be used for getting notifications on an > update to any node. > > Regards, > Mahendra > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > This feature is tricky to implement because elasticsearch is built quite > > differently than couchdb, also, couchdb has it easy since its a single node. > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) > > wrote: > > > > Yes , i mean the \_change API. > > > ([http://guide.couchdb.org/draft/notifications.html-](http://guide.couchdb.org/draft/notifications.html-) Continious > > > notification) > > > > I am sure this feature will help many ppl in their integration process. > > > > Thanks > > > Vineeth > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > There isn't an option for that. How do you envision the notification, as > > > > a pull or push service? With couchdb, do you refer to the \_changes API? > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > [VineethMo...@algotree.com](mailto:VineethMo...@algotree.com) wrote: > > > > > > In couchDB there is a service where any external application > > > > > interested in a particular document can hook to coucDB and if the sth > > > > > document is available , couchDB notifies the external application. > > > > > > Is there any similar feature in elasticSearch ? > > > > > Basically i have another service running somewhere and it is > > > > > interested in documents added to elasticSearch. > > > > > > Thanks > > > > > Vineeth > > -- > Mahendra > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 10, 2011, 2:16pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/10 "2011-08-10T14:16:01Z")
Yea, but that heavily relies on how couchdb works internally, which is very different than how elasticsearch works. On Wed, Aug 10, 2011 at 5:04 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > Hi Shay, > > CouchDB has distributed node support via BigCouch. They also have a > \_changes interface which can be used for getting notifications on an > update to any node. > > Regards, > Mahendra > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > This feature is tricky to implement because elasticsearch is built quite > > differently than couchdb, also, couchdb has it easy since its a single > > node. > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> > > wrote: > > > > > Yes , i mean the \_change API. > > > ([http://guide.couchdb.org/draft/notifications.html](http://guide.couchdb.org/draft/notifications.html) - Continious > > > notification) > > > > > > I am sure this feature will help many ppl in their integration process. > > > > > > Thanks > > > Vineeth > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > There isn't an option for that. How do you envision the notification, > > > > as > > > > a pull or push service? With couchdb, do you refer to the \_changes API? > > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > > > > > > > > > In couchDB there is a service where any external application > > > > > interested in a particular document can hook to coucDB and if the sth > > > > > document is available , couchDB notifies the external application. > > > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > Basically i have another service running somewhere and it is > > > > > interested in documents added to elasticSearch. > > > > > > > > > > Thanks > > > > > Vineeth > > -- > Mahendra > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 2:23pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/11 "2011-08-10T14:23:22Z")
Anyway when a document is added to a node , it notifies that to all other nodes. Cant we add a plugin point at the place where documents are accepted via HTTP or when replications comes. I am sure lot many people can use this plugin point to make custom plugins. Thanks Vineeth On Wed, Aug 10, 2011 at 7:46 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Yea, but that heavily relies on how couchdb works internally, which is very > different than how elasticsearch works. > > On Wed, Aug 10, 2011 at 5:04 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > > > Hi Shay, > > > > CouchDB has distributed node support via BigCouch. They also have a > > \_changes interface which can be used for getting notifications on an > > update to any node. > > > > Regards, > > Mahendra > > > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > This feature is tricky to implement because elasticsearch is built quite > > > differently than couchdb, also, couchdb has it easy since its a single > > > node. > > > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> > > > wrote: > > > > > > > Yes , i mean the \_change API. > > > > ([http://guide.couchdb.org/draft/notifications.html](http://guide.couchdb.org/draft/notifications.html) - Continious > > > > notification) > > > > > > > > I am sure this feature will help many ppl in their integration process. > > > > > > > > Thanks > > > > Vineeth > > > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > There isn't an option for that. How do you envision the notification, > > > > > as > > > > > a pull or push service? With couchdb, do you refer to the \_changes > > > > > API? > > > > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > > > > > > > > > > > In couchDB there is a service where any external application > > > > > > interested in a particular document can hook to coucDB and if the sth > > > > > > document is available , couchDB notifies the external application. > > > > > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > > Basically i have another service running somewhere and it is > > > > > > interested in documents added to elasticSearch. > > > > > > > > > > > > Thanks > > > > > > Vineeth > > > > -- > > Mahendra > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 10, 2011, 2:54pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/12 "2011-08-10T14:54:29Z")
The problem here is not hooking into the point where a document is added. Its more of how those notifications will be sent. Is it similar to the \_changes feed where you continuously get changes, and in this case, what happens if you want to get previous docs as well, or when the client being notified was disconnected and would like to get all the changes that happened while it was disconnected. Or, is it a push feed, and in this case, how to push those changes (transport type). What happens when the client is not there to receive the changes, or its just slow. The feature itself is much more heavyweight than what I think. On Wed, Aug 10, 2011 at 5:23 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Anyway when a document is added to a node , it notifies that to all other > nodes. > Cant we add a plugin point at the place where documents are accepted via > HTTP or when replications comes. > I am sure lot many people can use this plugin point to make custom plugins. > > Thanks > Vineeth > > On Wed, Aug 10, 2011 at 7:46 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Yea, but that heavily relies on how couchdb works internally, which is > > very different than how elasticsearch works. > > > > On Wed, Aug 10, 2011 at 5:04 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > > > > > Hi Shay, > > > > > > CouchDB has distributed node support via BigCouch. They also have a > > > \_changes interface which can be used for getting notifications on an > > > update to any node. > > > > > > Regards, > > > Mahendra > > > > > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > This feature is tricky to implement because elasticsearch is built > > > > quite > > > > differently than couchdb, also, couchdb has it easy since its a single > > > > node. > > > > > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> > > > > wrote: > > > > > > > > > Yes , i mean the \_change API. > > > > > ([http://guide.couchdb.org/draft/notifications.html](http://guide.couchdb.org/draft/notifications.html) - Continious > > > > > notification) > > > > > > > > > > I am sure this feature will help many ppl in their integration > > > > > process. > > > > > > > > > > Thanks > > > > > Vineeth > > > > > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > > > There isn't an option for that. How do you envision the notification, > > > > > > as > > > > > > a pull or push service? With couchdb, do you refer to the \_changes > > > > > > API? > > > > > > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > > > > > > > > > > > > > In couchDB there is a service where any external application > > > > > > > interested in a particular document can hook to coucDB and if the > > > > > > > sth > > > > > > > document is available , couchDB notifies the external application. > > > > > > > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > > > Basically i have another service running somewhere and it is > > > > > > > interested in documents added to elasticSearch. > > > > > > > > > > > > > > Thanks > > > > > > > Vineeth > > > > > > -- > > > Mahendra > > > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 3:20pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/13 "2011-08-10T15:20:32Z")
Is there a global revision number like we have in SVN in elasticSearch. If there is , the problem Shay is speaking can be resolved. Like when there is a disconnection if the revision number was 210 , again when it is connected , it can demand all events from revision number 210 to current. I believe the global revision number can be implemented in the plugin itself. I wonder how this is handled in couchDB !!! Mahendra , can you shed some insigh into this. Say in couchDB , while doing contious notifications , it so happened that the the connection hook got lost. How will the external application catch up with the changes when the network hook is established again ... Thanks Vineeth On Wed, Aug 10, 2011 at 8:24 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > The problem here is not hooking into the point where a document is added. > Its more of how those notifications will be sent. > > Is it similar to the \_changes feed where you continuously get changes, and > in this case, what happens if you want to get previous docs as well, or when > the client being notified was disconnected and would like to get all the > changes that happened while it was disconnected. > > Or, is it a push feed, and in this case, how to push those changes > (transport type). What happens when the client is not there to receive the > changes, or its just slow. > > The feature itself is much more heavyweight than what I think. > > On Wed, Aug 10, 2011 at 5:23 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Anyway when a document is added to a node , it notifies that to all other > > nodes. > > Cant we add a plugin point at the place where documents are accepted via > > HTTP or when replications comes. > > I am sure lot many people can use this plugin point to make custom > > plugins. > > > > Thanks > > Vineeth > > > > On Wed, Aug 10, 2011 at 7:46 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > Yea, but that heavily relies on how couchdb works internally, which is > > > very different than how elasticsearch works. > > > > > > On Wed, Aug 10, 2011 at 5:04 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com)wrote: > > > > > > > Hi Shay, > > > > > > > > CouchDB has distributed node support via BigCouch. They also have a > > > > \_changes interface which can be used for getting notifications on an > > > > update to any node. > > > > > > > > Regards, > > > > Mahendra > > > > > > > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > This feature is tricky to implement because elasticsearch is built > > > > > quite > > > > > differently than couchdb, also, couchdb has it easy since its a single > > > > > node. > > > > > > > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> > > > > > wrote: > > > > > > > > > > > Yes , i mean the \_change API. > > > > > > ([Change Notifications](http://guide.couchdb.org/draft/notifications.html) - Continious > > > > > > notification) > > > > > > > > > > > > I am sure this feature will help many ppl in their integration > > > > > > process. > > > > > > > > > > > > Thanks > > > > > > Vineeth > > > > > > > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > There isn't an option for that. How do you envision the > > > > > > > notification, as > > > > > > > a pull or push service? With couchdb, do you refer to the \_changes > > > > > > > API? > > > > > > > > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > > > > > > > > > > > > > > > In couchDB there is a service where any external application > > > > > > > > interested in a particular document can hook to coucDB and if the > > > > > > > > sth > > > > > > > > document is available , couchDB notifies the external application. > > > > > > > > > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > > > > Basically i have another service running somewhere and it is > > > > > > > > interested in documents added to elasticSearch. > > > > > > > > > > > > > > > > Thanks > > > > > > > > Vineeth > > > > > > > > -- > > > > Mahendra > > > > > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 10, 2011, 3:45pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/14 "2011-08-10T15:45:25Z")
couchDB have something similar cocept to the revision number in source control likes SVN. It maintains a revision number for each database. Hence we can do a curl to couchDB like curl -X GET $HOST/db/\_changes?since=REVISION\_NUMBER It will only fetch the results from the last REVISION\_NUMBER. Let me know you thoughts , if you feel this idea is workable , it will help lots of folks out there. Thanks Vineeth On Wed, Aug 10, 2011 at 8:50 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Is there a global revision number like we have in SVN in elasticSearch. > If there is , the problem Shay is speaking can be resolved. > Like when there is a disconnection if the revision number was 210 , again > when it is connected , it can demand all events from revision number 210 to > current. > I believe the global revision number can be implemented in the plugin > itself. > > I wonder how this is handled in couchDB !!! > Mahendra , can you shed some insigh into this. Say in couchDB , while doing > contious notifications , it so happened that the the connection hook got > lost. > How will the external application catch up with the changes when the > network hook is established again ... > > Thanks > Vineeth > > On Wed, Aug 10, 2011 at 8:24 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > The problem here is not hooking into the point where a document is added. > > Its more of how those notifications will be sent. > > > > Is it similar to the \_changes feed where you continuously get changes, and > > in this case, what happens if you want to get previous docs as well, or when > > the client being notified was disconnected and would like to get all the > > changes that happened while it was disconnected. > > > > Or, is it a push feed, and in this case, how to push those changes > > (transport type). What happens when the client is not there to receive the > > changes, or its just slow. > > > > The feature itself is much more heavyweight than what I think. > > > > On Wed, Aug 10, 2011 at 5:23 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) > > > > > wrote: > > > > > Anyway when a document is added to a node , it notifies that to all other > > > nodes. > > > Cant we add a plugin point at the place where documents are accepted via > > > HTTP or when replications comes. > > > I am sure lot many people can use this plugin point to make custom > > > plugins. > > > > > > Thanks > > > Vineeth > > > > > > On Wed, Aug 10, 2011 at 7:46 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > Yea, but that heavily relies on how couchdb works internally, which is > > > > very different than how elasticsearch works. > > > > > > > > On Wed, Aug 10, 2011 at 5:04 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com)wrote: > > > > > > > > > Hi Shay, > > > > > > > > > > CouchDB has distributed node support via BigCouch. They also have a > > > > > \_changes interface which can be used for getting notifications on an > > > > > update to any node. > > > > > > > > > > Regards, > > > > > Mahendra > > > > > > > > > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > > > This feature is tricky to implement because elasticsearch is built > > > > > > quite > > > > > > differently than couchdb, also, couchdb has it easy since its a > > > > > > single node. > > > > > > > > > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> > > > > > > wrote: > > > > > > > > > > > > > Yes , i mean the \_change API. > > > > > > > ([http://guide.couchdb.org/draft/notifications.html](http://guide.couchdb.org/draft/notifications.html) - Continious > > > > > > > notification) > > > > > > > > > > > > > > I am sure this feature will help many ppl in their integration > > > > > > > process. > > > > > > > > > > > > > > Thanks > > > > > > > Vineeth > > > > > > > > > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > > > > > > > wrote: > > > > > > > > > > > > > > > There isn't an option for that. How do you envision the > > > > > > > > notification, as > > > > > > > > a pull or push service? With couchdb, do you refer to the \_changes > > > > > > > > API? > > > > > > > > > > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > > > > > [VineethMohan@algotree.com](mailto:VineethMohan@algotree.com) wrote: > > > > > > > > > > > > > > > > > In couchDB there is a service where any external application > > > > > > > > > interested in a particular document can hook to coucDB and if the > > > > > > > > > sth > > > > > > > > > document is available , couchDB notifies the external application. > > > > > > > > > > > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > > > > > Basically i have another service running somewhere and it is > > > > > > > > > interested in documents added to elasticSearch. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Vineeth > > > > > > > > > > -- > > > > > Mahendra > > > > > > > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [August 10, 2011, 4:57pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/15 "2011-08-10T16:57:49Z")
Btw. does ES have something like Solr's UpdateRequestProcessor? I think the answer is negative, but if it had/has that, would that be a good place to plug in custom "new document listeners"? ## Otis Sematext is hiring Search Engineers -- [Jobs](http://sematext.com/about/jobs.html) On Aug 10, 11:45 am, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > couchDB have something similar cocept to the revision number in source > control likes SVN. > It maintains a revision number for each database. > Hence we can do a curl to couchDB like > > curl -X GET $HOST/db/\_changes?since=REVISION\_NUMBER > > It will only fetch the results from the last REVISION\_NUMBER. > > Let me know you thoughts , if you feel this idea is workable , it will help > lots of folks out there. > > Thanks > Vineeth > > On Wed, Aug 10, 2011 at 8:50 PM, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com)wrote: > > > Is there a global revision number like we have in SVN in elasticSearch. > > If there is , the problem Shay is speaking can be resolved. > > Like when there is a disconnection if the revision number was 210 , again > > when it is connected , it can demand all events from revision number 210 to > > current. > > I believe the global revision number can be implemented in the plugin > > itself. > > > I wonder how this is handled in couchDB !!! > > Mahendra , can you shed some insigh into this. Say in couchDB , while doing > > contious notifications , it so happened that the the connection hook got > > lost. > > How will the external application catch up with the changes when the > > network hook is established again ... > > > Thanks > > Vineeth > > > On Wed, Aug 10, 2011 at 8:24 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > The problem here is not hooking into the point where a document is added. > > > Its more of how those notifications will be sent. > > > > Is it similar to the \_changes feed where you continuously get changes, and > > > in this case, what happens if you want to get previous docs as well, or when > > > the client being notified was disconnected and would like to get all the > > > changes that happened while it was disconnected. > > > > Or, is it a push feed, and in this case, how to push those changes > > > (transport type). What happens when the client is not there to receive the > > > changes, or its just slow. > > > > The feature itself is much more heavyweight than what I think. > > > > On Wed, Aug 10, 2011 at 5:23 PM, Vineeth Mohan \<[vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) > > > > > > > wrote: > > > > > Anyway when a document is added to a node , it notifies that to all other > > > > nodes. > > > > Cant we add a plugin point at the place where documents are accepted via > > > > HTTP or when replications comes. > > > > I am sure lot many people can use this plugin point to make custom > > > > plugins. > > > > > Thanks > > > > Vineeth > > > > > On Wed, Aug 10, 2011 at 7:46 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > Yea, but that heavily relies on how couchdb works internally, which is > > > > > very different than how elasticsearch works. > > > > > > On Wed, Aug 10, 2011 at 5:04 PM, Mahendra M [mahendr...@gmail.com](mailto:mahendr...@gmail.com)wrote: > > > > > > > Hi Shay, > > > > > > > CouchDB has distributed node support via BigCouch. They also have a > > > > > > \_changes interface which can be used for getting notifications on an > > > > > > update to any node. > > > > > > > Regards, > > > > > > Mahendra > > > > > > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > > > > > > > This feature is tricky to implement because elasticsearch is built > > > > > > > quite > > > > > > > differently than couchdb, also, couchdb has it easy since its a > > > > > > > single node. > > > > > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > > > > > > [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com)\> > > > > > > > wrote: > > > > > > > > > Yes , i mean the \_change API. > > > > > > > > ([http://guide.couchdb.org/draft/notifications.html-](http://guide.couchdb.org/draft/notifications.html-) Continious > > > > > > > > notification) > > > > > > > > > I am sure this feature will help many ppl in their integration > > > > > > > > process. > > > > > > > > > Thanks > > > > > > > > Vineeth > > > > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) > > > > > > > > wrote: > > > > > > > > > > There isn't an option for that. How do you envision the > > > > > > > > > notification, as > > > > > > > > > a pull or push service? With couchdb, do you refer to the \_changes > > > > > > > > > API? > > > > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan > > > > > > > > > [VineethMo...@algotree.com](mailto:VineethMo...@algotree.com) wrote: > > > > > > > > > > > In couchDB there is a service where any external application > > > > > > > > > > interested in a particular document can hook to coucDB and if the > > > > > > > > > > sth > > > > > > > > > > document is available , couchDB notifies the external application. > > > > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > > > > > > Basically i have another service running somewhere and it is > > > > > > > > > > interested in documents added to elasticSearch. > > > > > > > > > > > Thanks > > > > > > > > > > Vineeth > > > > > > > -- > > > > > > Mahendra > > > > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [August 10, 2011, 6:15pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/16 "2011-08-10T18:15:46Z")
In my opinion, you can use percolation in a similar manner to filtered CouchDB \_changes feed: you'll receive matching queries for every document being indexed. On Aug 10, 12:32 pm, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > If someone can give tips and clue on how o implement this , i am willing to > make a open source plugin out of this. > > Thanks > Vineeth > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > This feature is tricky to implement because elasticsearch is built quite > > differently than couchdb, also, couchdb has it easy since its a single node. > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com)wrote: > > > > Yes , i mean the \_change API. > > > ([http://guide.couchdb.org/draft/notifications.html-](http://guide.couchdb.org/draft/notifications.html-) Continious > > > notification) > > > > I am sure this feature will help many ppl in their integration process. > > > > Thanks > > > Vineeth > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > There isn't an option for that. How do you envision the notification, as > > > > a pull or push service? With couchdb, do you refer to the \_changes API? > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan \< > > > > [VineethMo...@algotree.com](mailto:VineethMo...@algotree.com)\> wrote: > > > > > > In couchDB there is a service where any external application > > > > > interested in a particular document can hook to coucDB and if the sth > > > > > document is available , couchDB notifies the external application. > > > > > > Is there any similar feature in elasticSearch ? > > > > > Basically i have another service running somewhere and it is > > > > > interested in documents added to elasticSearch. > > > > > > Thanks > > > > > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 11, 2011, 9:57am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/17 "2011-08-11T09:57:54Z")
Hello Karmi , Can you explain how to use percolation to achieve the feature i want ???. My whole idea is to decouple the module that indexes the document and the external applications that are interested in new documents. AFAIK peroclation provides a method to reverse map queries. That is a query can be registered and when a new document comes , percolator tells which all queries were registed. How can this feature be used by an external application to determine if a new document is indexed. The feature i am looking is well implemented in couchDB. If you have any doubt please go through - [http://guide.couchdb.org/draft/notifications.html](http://guide.couchdb.org/draft/notifications.html) They have something like the global revision number as seen in source code repo's like SVN. On each poll , the external application can tell ES to give all document that were added/edited since thi revision number. Thanks Vineeth On Wed, Aug 10, 2011 at 11:45 PM, karmi [karel.minarik@gmail.com](mailto:karel.minarik@gmail.com) wrote: > In my opinion, you can use percolation in a similar manner to filtered > CouchDB \_changes feed: you'll receive matching queries for every > document being indexed. > > On Aug 10, 12:32 pm, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > > > If someone can give tips and clue on how o implement this , i am willing > > to > > make a open source plugin out of this. > > > > Thanks > > Vineeth > > > > On Wed, Aug 10, 2011 at 3:59 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > This feature is tricky to implement because elasticsearch is built > > > quite > > > differently than couchdb, also, couchdb has it easy since its a single > > > node. > > > > > On Wed, Aug 10, 2011 at 1:28 PM, Vineeth Mohan \< > > > [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com)\>wrote: > > > > > > Yes , i mean the \_change API. > > > > ([http://guide.couchdb.org/draft/notifications.html-](http://guide.couchdb.org/draft/notifications.html-) Continious > > > > notification) > > > > > > I am sure this feature will help many ppl in their integration > > > > process. > > > > > > Thanks > > > > Vineeth > > > > > > On Wed, Aug 10, 2011 at 3:51 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > There isn't an option for that. How do you envision the notification, > > > > > as > > > > > a pull or push service? With couchdb, do you refer to the \_changes > > > > > API? > > > > > > > On Wed, Aug 10, 2011 at 9:16 AM, Vineeth Mohan \< > > > > > [VineethMo...@algotree.com](mailto:VineethMo...@algotree.com)\> wrote: > > > > > > > > In couchDB there is a service where any external application > > > > > > interested in a particular document can hook to coucDB and if the > > > > > > sth > > > > > > document is available , couchDB notifies the external application. > > > > > > > > Is there any similar feature in elasticSearch ? > > > > > > Basically i have another service running somewhere and it is > > > > > > interested in documents added to elasticSearch. > > > > > > > > Thanks > > > > > > Vineeth ---
**Author:** ![David\_Richardson](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@David\_Richardson](https://discuss.elastic.co/u/David_Richardson)\ **Post date:** [August 13, 2011, 5:30pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/18 "2011-08-13T17:30:21Z")
How then would one push change events into rabbitmq, or some other message broker. Not my preferred mechanism, since rabbitmq isn't distributed, but perhaps that isn't so important for change events. Soft realtime required, polling not allowed. Doing this "in the (external) app" isn't viable. Change notifications and WAN replication are really the only things missing in es that preclude decommissioning our couchdb infrastructure - which we would like to do since virtually every query against must already go through external search. Getting change events into an external message broker provides an immediate solution to both, but perhaps that's no easier than an internal changes feed. btw, Postgresql provides an even better model for external notifications[http://www.postgresql.org/docs/9.0/interactive/sql-notify.html](http://www.postgresql.org/docs/9.0/interactive/sql-notify.html)imho - multiple channels plus a programmable payload. Have no experience with it at extreme load, but under moderate load it works wonderfully. Again, radically different technical environment - it's the api model that's of interest. What we're talking about for ES is a river producer, rather than consumer. cheers, d.r. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 13, 2011, 7:35pm UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/19 "2011-08-13T19:35:24Z")
Let me do a quick brain dump here, and try to explain what needs to be done to properly support this: First, one can (with my help, or looking at the code) write a plugin that registers for indexing operations. The listener can also check and make sure to only process events that happen on a primary shard (so they won't be processed on the replica, if a design requires it). But, to be honest, this is the easy part. For changes feed, one has two options, pull and push. Lets start with push. Push notifications are quire simple to implement in a non distributed solution (like redis does). People register listeners and every time an operation happens, the listeners get notified. It does require some thought as to how to publish those events. If one controls the clients as well, then its simple (i.e. doing it only for the Java API), but, since elasticsearch treats HTTP as a first class citizen, a solution for HTTP needs to be built as well. This can be similar to pusspubsub...( [PubSubHubbub · GitHub](http://code.google.com/p/pubsubhubbub/)), but then the clients needs also to listen for HTTP requests. Also, with push notifications, there is a question of do we only send "new" events to the listener, or also send all the current data (possibly filtered) and later, new events happening. There is a question of what to do with misbehaving endpoints that don't process notifications fast enough (tricky to identify it...), block them, drop them, or something similar. Also, there is a question if the listeners are persistent. If they go away, do we queue events and send it to them once they reconnect? Now, lets move to a distributed solution. Lets start with simple HA, replication. Now, we need to make sure that listeners registrations are persisted across the cluster (and possibly surviving full cluster restart). Also, we need to make sure as shards move around that those listeners move around with them. Also, if we support persistent notifications, we need the queue of future events that need to be sent to disconnected clients is replicated as well (and we need to recover them, get this data into hot relocation of shards, and so on). Now, lets talk about pull notifications, which is similar to how couchdb does things. First, a note on couchdb. The data structure couchdb has (basically, a never ending (up to compaction) btree) is a big boon when it comes to implementing pull notifications. elasticsearch/lucene do not work like that. Pull notification will probably require API based invocation of give me changes since X. X can be a timestamp, or an id that denotes some sort of "timeability"/order. A user will need to register the fact that it starts listener, and we in elasticsearch can make sure that any changes are kept around for the next pull request the user does (either on an open HTTP connection, or per request, does not really matter). This is a bit simpler to implement in elasticsearch, we can keep the transaction log around long enough till we notified all clients about the changes, and, it allows us to do async notifications more easily. But, it still requires delicate control over the transaction log and when we can safely "get rid" of it. Also, pull notification require thought as to how to provide all the "current" data in elasticsearch, Again, its certainly possible, and the user can provide a query that will filter that data if not all data is needed. In terms of the internals of how elasticsearch works, pull notification is simpler, but still require delicate work when it comes to concurrency, transaction log handling, that are pretty low level... . Not simple. Summary: One of the things left on the plate for elasticsearch is cross data center replication. I would love to implement it in a way that cross data center replication mechanism is open enough for users to use. What does it mean? For example, if we do pull based notifications, we can possibly utilize that for cross data center replication. Another cluster, halfway around the world, is just another user of the pull based notifications. Hope things make a bit more sense now... 🙂 On Sat, Aug 13, 2011 at 8:30 PM, David Richardson \< [david.richardson@enquora.com](mailto:david.richardson@enquora.com)\> wrote: > How then would one push change events into rabbitmq, or some other message > broker. Not my preferred mechanism, since rabbitmq isn't distributed, but > perhaps that isn't so important for change events. Soft realtime required, > polling not allowed. Doing this "in the (external) app" isn't viable. > > Change notifications and WAN replication are really the only things missing > in es that preclude decommissioning our couchdb infrastructure - which we > would like to do since virtually every query against must already go through > external search. Getting change events into an external message broker > provides an immediate solution to both, but perhaps that's no easier than an > internal changes feed. > > btw, Postgresql provides an even better model for external notifications[http://www.postgresql.org/docs/9.0/interactive/sql-notify.html](http://www.postgresql.org/docs/9.0/interactive/sql-notify.html)imho - multiple channels plus a programmable payload. Have no experience > with it at extreme load, but under moderate load it works wonderfully. > Again, radically different technical environment - it's the api model that's > of interest. What we're talking about for ES is a river producer, rather > than consumer. > > cheers, > d.r. ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [August 15, 2011, 11:19am UTC](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106/20 "2011-08-15T11:19:57Z")
Thanks for taking up time to provide all your thoughts. I am happy that you are thinking the same way we do , on the problem. Once we can make this feature , on writing a ES river we can replicate ES over to another cluster. Also i have made a feature request on the same - > > > There should be an integration point for ES and external application where the e…xternal applications should be notified of any document changes or updates that happens in ES. > > CouchDB have a good implementation on it and it would be great if ES can also incorporate something similar or same. > > CouchDB change notification feature - http://guide.couchdb.org/draft/notifications.html Thanks Vineeth On Sun, Aug 14, 2011 at 1:05 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Let me do a quick brain dump here, and try to explain what needs to be done > to properly support this: > > First, one can (with my help, or looking at the code) write a plugin that > registers for indexing operations. The listener can also check and make sure > to only process events that happen on a primary shard (so they won't be > processed on the replica, if a design requires it). But, to be honest, this > is the easy part. > > For changes feed, one has two options, pull and push. > > Lets start with push. Push notifications are quire simple to implement in a > non distributed solution (like redis does). People register listeners and > every time an operation happens, the listeners get notified. It does require > some thought as to how to publish those events. If one controls the clients > as well, then its simple (i.e. doing it only for the Java API), but, since > elasticsearch treats HTTP as a first class citizen, a solution for HTTP > needs to be built as well. This can be similar to pusspubsub...( > [PubSubHubbub · GitHub](http://code.google.com/p/pubsubhubbub/)), but then the clients needs also > to listen for HTTP requests. > > Also, with push notifications, there is a question of do we only send "new" > events to the listener, or also send all the current data (possibly > filtered) and later, new events happening. There is a question of what to do > with misbehaving endpoints that don't process notifications fast enough > (tricky to identify it...), block them, drop them, or something similar. > > Also, there is a question if the listeners are persistent. If they go away, > do we queue events and send it to them once they reconnect? > > Now, lets move to a distributed solution. Lets start with simple HA, > replication. Now, we need to make sure that listeners registrations are > persisted across the cluster (and possibly surviving full cluster restart). > Also, we need to make sure as shards move around that those listeners move > around with them. Also, if we support persistent notifications, we need the > queue of future events that need to be sent to disconnected clients is > replicated as well (and we need to recover them, get this data into hot > relocation of shards, and so on). > > Now, lets talk about pull notifications, which is similar to how couchdb > does things. First, a note on couchdb. The data structure couchdb has > (basically, a never ending (up to compaction) btree) is a big boon when it > comes to implementing pull notifications. elasticsearch/lucene do not work > like that. > > Pull notification will probably require API based invocation of give me > changes since X. X can be a timestamp, or an id that denotes some sort of > "timeability"/order. A user will need to register the fact that it starts > listener, and we in elasticsearch can make sure that any changes are kept > around for the next pull request the user does (either on an open HTTP > connection, or per request, does not really matter). This is a bit simpler > to implement in elasticsearch, we can keep the transaction log around long > enough till we notified all clients about the changes, and, it allows us to > do async notifications more easily. But, it still requires delicate control > over the transaction log and when we can safely "get rid" of it. > > Also, pull notification require thought as to how to provide all the > "current" data in elasticsearch, Again, its certainly possible, and the user > can provide a query that will filter that data if not all data is needed. > > In terms of the internals of how elasticsearch works, pull notification is > simpler, but still require delicate work when it comes to concurrency, > transaction log handling, that are pretty low level... . Not simple. > > Summary: > > One of the things left on the plate for elasticsearch is cross data center > replication. I would love to implement it in a way that cross data center > replication mechanism is open enough for users to use. What does it mean? > For example, if we do pull based notifications, we can possibly utilize that > for cross data center replication. Another cluster, halfway around the > world, is just another user of the pull based notifications. > > Hope things make a bit more sense now... 🙂 > > On Sat, Aug 13, 2011 at 8:30 PM, David Richardson \< > [david.richardson@enquora.com](mailto:david.richardson@enquora.com)\> wrote: > > > How then would one push change events into rabbitmq, or some other message > > broker. Not my preferred mechanism, since rabbitmq isn't distributed, but > > perhaps that isn't so important for change events. Soft realtime required, > > polling not allowed. Doing this "in the (external) app" isn't viable. > > > > Change notifications and WAN replication are really the only things > > missing in es that preclude decommissioning our couchdb infrastructure - > > which we would like to do since virtually every query against must already > > go through external search. Getting change events into an external message > > broker provides an immediate solution to both, but perhaps that's no easier > > than an internal changes feed. > > > > btw, Postgresql provides an even better model for external notifications[http://www.postgresql.org/docs/9.0/interactive/sql-notify.html](http://www.postgresql.org/docs/9.0/interactive/sql-notify.html)imho - multiple channels plus a programmable payload. Have no experience > > with it at extreme load, but under moderate load it works wonderfully. > > Again, radically different technical environment - it's the api model that's > > of interest. What we're talking about for ES is a river producer, rather > > than consumer. > > > > cheers, > > d.r. [Next page](https://discuss.elastic.co/t/notifications-from-elasticsearch-when-documents-are-added/5106.md?page=2)