# Using elasticsearch as a realtime fire hose **URL:** \ **Category:** Elasticsearch\ **Created:** [August 23, 2014, 6:09pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-a-realtime-fire-hose/19421 "2014-08-23T18:09:14Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jim\_Alateras](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jim_alateras/32/1327_2.png) [@Jim\_Alateras](https://discuss.elastic.co/u/Jim_Alateras)\ **Post date:** [August 23, 2014, 6:09pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-a-realtime-fire-hose/19421/1 "2014-08-23T18:09:14Z")
I was wondering whether there were any mechanisms to use ES as a realtime feed for downstream systems. I have a cluster that gathers observations from many sensors. I have a need to maintain a list of realtime counters in REDIS so I want to further process these observation once they hit the database. Additionally I also want to be able to create event streams for different type of feeds. I could do all this outside ES but I was wondering whether there were mechanisms within ES that will allow me to subscribe to add events for a particular type or index. cheers -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 23, 2014, 8:04pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-a-realtime-fire-hose/19421/2 "2014-08-23T20:04:32Z")
What kind of events do you think of? Single new document indexed? Batch of docs indexed? Node-wide? Or cluster wide? You mention Redis, for something like publish/subscribe pattern, you'd have to use a persistent connection and implement your own ES actions, which is possible with e.g. HTTP websockets A sketchy implementation can be found here: > **[GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket)** > > WebSockets for ElasticSearch. Contribute to jprante/elasticsearch-transport-websocket development by creating an account on GitHub. Jörg On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras [jim@sutoiku.com](mailto:jim@sutoiku.com) wrote: > I was wondering whether there were any mechanisms to use ES as a realtime > feed for downstream systems. I have a cluster that gathers observations > from many sensors. I have a need to maintain a list of realtime counters in > REDIS so I want to further process these observation once they hit the > database. Additionally I also want to be able to create event streams for > different type of feeds. > > I could do all this outside ES but I was wondering whether there were > mechanisms within ES that will allow me to subscribe to add events for a > particular type or index. > > cheers > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEkBLfd1pcQ%3D0hf6fstV7oJvO%2BB2GwF%3DdRyNqcN0%3D%2BNzw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEkBLfd1pcQ%3D0hf6fstV7oJvO%2BB2GwF%3DdRyNqcN0%3D%2BNzw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jim\_Alateras](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jim_alateras/32/1327_2.png) [@Jim\_Alateras](https://discuss.elastic.co/u/Jim_Alateras)\ **Post date:** [August 25, 2014, 7:27am UTC](https://discuss.elastic.co/t/using-elasticsearch-as-a-realtime-fire-hose/19421/3 "2014-08-25T07:27:42Z")
> What kind of events do you think of? Single new document indexed? Batch of > docs indexed? Node-wide? Or cluster wide? event on whenever a document is added to an index cluster wide > You mention Redis, for something like publish/subscribe pattern, you'd > have to use a persistent connection and implement your own ES actions, > which is possible with e.g. HTTP websockets > > A sketchy implementation can be found here: > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) thanks for the reference, I will have a deeper look at it. > Jörg > > On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras \<[j...@sutoiku.com](mailto:j...@sutoiku.com) > \\> wrote: > > > I was wondering whether there were any mechanisms to use ES as a realtime > > feed for downstream systems. I have a cluster that gathers observations > > from many sensors. I have a need to maintain a list of realtime counters in > > REDIS so I want to further process these observation once they hit the > > database. Additionally I also want to be able to create event streams for > > different type of feeds. > > > > I could do all this outside ES but I was wondering whether there were > > mechanisms within ES that will allow me to subscribe to add events for a > > particular type or index. > > > > cheers > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7a816e7f-2a58-49ed-8188-08d4483bc2d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7a816e7f-2a58-49ed-8188-08d4483bc2d2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [August 26, 2014, 2:28pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-a-realtime-fire-hose/19421/4 "2014-08-26T14:28:12Z")
You might want to look at developing a plugin for this or maybe using an existing one. This one for example might do partly what you need: [GitHub - derryx/elasticsearch-changes-plugin: An Elasticsearch Plugin that notifies about changes to indices](https://github.com/derryx/elasticsearch-changes-plugin) If you develop your own plugin, you should be able to tap into what is happening in the cluster at a pretty low level. Jilles On Monday, August 25, 2014 9:27:42 AM UTC+2, Jim Alateras wrote: > What kind of events do you think of? Single new document indexed? Batch of > > > docs indexed? Node-wide? Or cluster wide? > > event on whenever a document is added to an index cluster wide > > > You mention Redis, for something like publish/subscribe pattern, you'd > > have to use a persistent connection and implement your own ES actions, > > which is possible with e.g. HTTP websockets > > > > A sketchy implementation can be found here: > > > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) > > thanks for the reference, I will have a deeper look at it. > > > Jörg > > > > On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras [j...@sutoiku.com](mailto:j...@sutoiku.com) wrote: > > > > > I was wondering whether there were any mechanisms to use ES as a > > > realtime feed for downstream systems. I have a cluster that gathers > > > observations from many sensors. I have a need to maintain a list of > > > realtime counters in REDIS so I want to further process these observation > > > once they hit the database. Additionally I also want to be able to create > > > event streams for different type of feeds. > > > > > > I could do all this outside ES but I was wondering whether there were > > > mechanisms within ES that will allow me to subscribe to add events for a > > > particular type or index. > > > > > > cheers > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3468dc3-2b96-4f00-a921-fba6892b5bba%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3468dc3-2b96-4f00-a921-fba6892b5bba%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/using-elasticsearch-as-a-realtime-fire-hose/19421/5 "2017-07-06T01:06:21Z")