I was wondering whether there were any mechanisms to use ES as a realtime
feed for downstream systems. I have a cluster that gathers observations
from many sensors. I have a need to maintain a list of realtime counters in
REDIS so I want to further process these observation once they hit the
database. Additionally I also want to be able to create event streams for
different type of feeds.
I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.
What kind of events do you think of? Single new document indexed? Batch of
docs indexed? Node-wide? Or cluster wide?
You mention Redis, for something like publish/subscribe pattern, you'd have
to use a persistent connection and implement your own ES actions, which is
possible with e.g. HTTP websockets
A sketchy implementation can be found here:
Jörg
On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras jim@sutoiku.com wrote:
I was wondering whether there were any mechanisms to use ES as a realtime
feed for downstream systems. I have a cluster that gathers observations
from many sensors. I have a need to maintain a list of realtime counters in
REDIS so I want to further process these observation once they hit the
database. Additionally I also want to be able to create event streams for
different type of feeds.
I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.
What kind of events do you think of? Single new document indexed? Batch of
docs indexed? Node-wide? Or cluster wide?
event on whenever a document is added to an index cluster wide
You mention Redis, for something like publish/subscribe pattern, you'd
have to use a persistent connection and implement your own ES actions,
which is possible with e.g. HTTP websockets
thanks for the reference, I will have a deeper look at it.
Jörg
On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras <j...@sutoiku.com
<javascript:>> wrote:
I was wondering whether there were any mechanisms to use ES as a realtime
feed for downstream systems. I have a cluster that gathers observations
from many sensors. I have a need to maintain a list of realtime counters in
REDIS so I want to further process these observation once they hit the
database. Additionally I also want to be able to create event streams for
different type of feeds.
I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.
If you develop your own plugin, you should be able to tap into what is
happening in the cluster at a pretty low level.
Jilles
On Monday, August 25, 2014 9:27:42 AM UTC+2, Jim Alateras wrote:
What kind of events do you think of? Single new document indexed? Batch of
docs indexed? Node-wide? Or cluster wide?
event on whenever a document is added to an index cluster wide
You mention Redis, for something like publish/subscribe pattern, you'd
have to use a persistent connection and implement your own ES actions,
which is possible with e.g. HTTP websockets
thanks for the reference, I will have a deeper look at it.
Jörg
On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras j...@sutoiku.com wrote:
I was wondering whether there were any mechanisms to use ES as a
realtime feed for downstream systems. I have a cluster that gathers
observations from many sensors. I have a need to maintain a list of
realtime counters in REDIS so I want to further process these observation
once they hit the database. Additionally I also want to be able to create
event streams for different type of feeds.
I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.