# How to stream results on a websocket with percolator? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2012, 1:53pm UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341 "2012-07-06T13:53:26Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Sebastien\_Lorber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastien_lorber/32/2551_2.png) [@Sebastien\_Lorber](https://discuss.elastic.co/u/Sebastien_Lorber)\ **Post date:** [July 6, 2012, 1:53pm UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341/1 "2012-07-06T13:53:26Z")
Hello What i would like to do is: - User1 and User2 are friend - Both have an open websocket - User1 post a new document - Because of friendship, User2 receives the document within the websocket I've checked the percolator. It seems i can use it that way: - User2 registers a percolator query to get documents of all its friends - User1 index the document and get the percolator query match (i think i read we can get percolator results at index time) - User1 current indexing sends notifications to its friends open websockets so that they receive the information The matter is that it can be quite expensive to send the notification from user1 doc indexing process to the user2 websocket handler. In a distributed webapp, it would require messaging. I think a tool which handles distributed event processing, like Redis, would fit for that need. To avoid using Redis or something else, would it be possible for ElasticSearch percolator to directly send the notification to the user2 webapp server. Is it possible that user2 registered percolator query keeps some kind of connection to elasticsearch and receive results as they get indexed? Thanks ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 6, 2012, 10:50pm UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341/2 "2012-07-06T22:50:13Z")
Hi, the percolator feature is fascinating, yes. Finding matches need another query action. The actions \_percolator and \_percolate are de-coupled, that is, it is not a publish/subscribe message passing model. See also the discussion for "change notification" at > > > There should be an integration point for ES and external application where the e…xternal applications should be notified of any document changes or updates that happens in ES. > > CouchDB have a good implementation on it and it would be great if ES can also incorporate something similar or same. > > CouchDB change notification feature - http://guide.couchdb.org/draft/notifications.html As far as I have examined the code of netty HTTP and REST in Elasticsearch, a publish/subscribe mechanism is really worth experimenting. I'm very interested in implementing such a beast. And, yes, I agree that Websockets are a good idea, because they are bi-directional and save resources. Imagine 10,000 clients waiting on a node for notification events. The scenario I'd like to explore is a curl websocket client "A" registering a 'tag' value on a specific index/type (using percolator behind the scenes) and hangs on waiting for events, while another curl client "B" is indexing. The change notifications are filtered and pushed to client "A" if they match until "A" closes the connection. Best regards, Jörg On Friday, July 6, 2012 3:53:26 PM UTC+2, Sébastien Lorber wrote: > Hello > > What i would like to do is: > > - User1 and User2 are friend > - Both have an open websocket > - User1 post a new document > - Because of friendship, User2 receives the document within the websocket > > I've checked the percolator. > > It seems i can use it that way: > > - User2 registers a percolator query to get documents of all its friends > - User1 index the document and get the percolator query match (i think i > read we can get percolator results at index time) > - User1 current indexing sends notifications to its friends open > websockets so that they receive the information > > The matter is that it can be quite expensive to send the notification from > user1 doc indexing process to the user2 websocket handler. > In a distributed webapp, it would require messaging. > I think a tool which handles distributed event processing, like Redis, > would fit for that need. > > To avoid using Redis or something else, would it be possible for > Elasticsearch percolator to directly send the notification to the user2 > webapp server. > Is it possible that user2 registered percolator query keeps some kind of > connection to elasticsearch and receive results as they get indexed? > > Thanks ---
**Author:** ![Sebastien\_Lorber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastien_lorber/32/2551_2.png) [@Sebastien\_Lorber](https://discuss.elastic.co/u/Sebastien_Lorber)\ **Post date:** [July 6, 2012, 11:46pm UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341/3 "2012-07-06T23:46:50Z")
Actually it's the idea but i'll use a Java backend instead of curl. I guess it's not so easy to implement since ES nodes should send notifications to each others about what's being indexed. As only one ES Netty should hold the websocket for one client, on large clusters it also means that if we have 100 nodes and only 1 client connected that should receive a notification, the notification should only be sent to the appropriate node and not the whole cluster. If you want to implement this i'm ok to help, don't know so much yet about websockets so it would be the occasion to learn... 2012/7/7 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > Hi, > > the percolator feature is fascinating, yes. Finding matches need another > query action. The actions \_percolator and \_percolate are de-coupled, that > is, it is not a publish/subscribe message passing model. > > See also the discussion for "change notification" at > [Changes API · Issue #1242 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1242) > > As far as I have examined the code of netty HTTP and REST in > Elasticsearch, a publish/subscribe mechanism is really worth experimenting. > I'm very interested in implementing such a beast. And, yes, I agree that > Websockets are a good idea, because they are bi-directional and save > resources. Imagine 10,000 clients waiting on a node for notification events. > > The scenario I'd like to explore is a curl websocket client "A" > registering a 'tag' value on a specific index/type (using percolator behind > the scenes) and hangs on waiting for events, while another curl client "B" > is indexing. The change notifications are filtered and pushed to client "A" > if they match until "A" closes the connection. > > Best regards, > > Jörg > > On Friday, July 6, 2012 3:53:26 PM UTC+2, Sébastien Lorber wrote: > > > Hello > > > > What i would like to do is: > > > > - User1 and User2 are friend > > - Both have an open websocket > > - User1 post a new document > > - Because of friendship, User2 receives the document within the websocket > > > > I've checked the percolator. > > > > It seems i can use it that way: > > > > - User2 registers a percolator query to get documents of all its friends > > - User1 index the document and get the percolator query match (i think i > > read we can get percolator results at index time) > > - User1 current indexing sends notifications to its friends open > > websockets so that they receive the information > > > > The matter is that it can be quite expensive to send the notification > > from user1 doc indexing process to the user2 websocket handler. > > In a distributed webapp, it would require messaging. > > I think a tool which handles distributed event processing, like Redis, > > would fit for that need. > > > > To avoid using Redis or something else, would it be possible for > > Elasticsearch percolator to directly send the notification to the user2 > > webapp server. > > Is it possible that user2 registered percolator query keeps some kind of > > connection to elasticsearch and receive results as they get indexed? > > > > Thanks ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 7, 2012, 12:52am UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341/4 "2012-07-07T00:52:00Z")
Sure, a pubsub architecture distributed across many ES nodes is not straightforward, it needs at least an internal pubsub index where the subscribers are identified and registered, together with the node location where the subscriber lives. Event classes must be designed. Classes like org.elasticsearch.action.TransportActionNodeProxy show me that actions can potentially be executed against a specific node, which is very useful for passing messages from the indexing node directly to the node where a subscriber waits for notification. An event buffer, also in the internal pubsub index, will also be useful, in situations where subscribers do not consume notifications as fast as they are produced. Websockets are also new to me but I'm encouraged by the netty examples... well I will need some time to move forward... Jörg On Saturday, July 7, 2012 1:46:50 AM UTC+2, Sébastien Lorber wrote: > Actually it's the idea but i'll use a Java backend instead of curl. > > I guess it's not so easy to implement since ES nodes should send > notifications to each others about what's being indexed. As only one ES > Netty should hold the websocket for one client, on large clusters it also > means that if we have 100 nodes and only 1 client connected that should > receive a notification, the notification should only be sent to the > appropriate node and not the whole cluster. > > If you want to implement this i'm ok to help, don't know so much yet about > websockets so it would be the occasion to learn... ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 7, 2012, 1:30pm UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341/5 "2012-07-07T13:30:22Z")
Note to self: Would be nice to have an API like the Hazelcast pubsub API [http://www.hazelcast.com/documentation.jsp#Topic](http://www.hazelcast.com/documentation.jsp#Topic) but just with Elasticsearch actions under the hood, with REST \_publish / \_subscribe endpoints, and indexing extended with setTopic() / \_topic field or something for automatic publishing while indexing. Jörg ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:21am UTC](https://discuss.elastic.co/t/how-to-stream-results-on-a-websocket-with-percolator/8341/6 "2017-07-06T03:21:09Z")