Using elasticsearch as a realtime fire hose


(Jim Alateras) #1

I was wondering whether there were any mechanisms to use ES as a realtime
feed for downstream systems. I have a cluster that gathers observations
from many sensors. I have a need to maintain a list of realtime counters in
REDIS so I want to further process these observation once they hit the
database. Additionally I also want to be able to create event streams for
different type of feeds.

I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.

cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

What kind of events do you think of? Single new document indexed? Batch of
docs indexed? Node-wide? Or cluster wide?

You mention Redis, for something like publish/subscribe pattern, you'd have
to use a persistent connection and implement your own ES actions, which is
possible with e.g. HTTP websockets

A sketchy implementation can be found here:

Jörg

On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras jim@sutoiku.com wrote:

I was wondering whether there were any mechanisms to use ES as a realtime
feed for downstream systems. I have a cluster that gathers observations
from many sensors. I have a need to maintain a list of realtime counters in
REDIS so I want to further process these observation once they hit the
database. Additionally I also want to be able to create event streams for
different type of feeds.

I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.

cheers

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEkBLfd1pcQ%3D0hf6fstV7oJvO%2BB2GwF%3DdRyNqcN0%3D%2BNzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jim Alateras) #3

What kind of events do you think of? Single new document indexed? Batch of
docs indexed? Node-wide? Or cluster wide?

event on whenever a document is added to an index cluster wide

You mention Redis, for something like publish/subscribe pattern, you'd
have to use a persistent connection and implement your own ES actions,
which is possible with e.g. HTTP websockets

A sketchy implementation can be found here:

https://github.com/jprante/elasticsearch-transport-websocket

thanks for the reference, I will have a deeper look at it.

Jörg

On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras <j...@sutoiku.com
<javascript:>> wrote:

I was wondering whether there were any mechanisms to use ES as a realtime
feed for downstream systems. I have a cluster that gathers observations
from many sensors. I have a need to maintain a list of realtime counters in
REDIS so I want to further process these observation once they hit the
database. Additionally I also want to be able to create event streams for
different type of feeds.

I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.

cheers

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7a816e7f-2a58-49ed-8188-08d4483bc2d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jilles van Gurp) #4

You might want to look at developing a plugin for this or maybe using an
existing one. This one for example might do partly what you
need: https://github.com/derryx/elasticsearch-changes-plugin

If you develop your own plugin, you should be able to tap into what is
happening in the cluster at a pretty low level.

Jilles

On Monday, August 25, 2014 9:27:42 AM UTC+2, Jim Alateras wrote:

What kind of events do you think of? Single new document indexed? Batch of

docs indexed? Node-wide? Or cluster wide?

event on whenever a document is added to an index cluster wide

You mention Redis, for something like publish/subscribe pattern, you'd
have to use a persistent connection and implement your own ES actions,
which is possible with e.g. HTTP websockets

A sketchy implementation can be found here:

https://github.com/jprante/elasticsearch-transport-websocket

thanks for the reference, I will have a deeper look at it.

Jörg

On Sat, Aug 23, 2014 at 8:09 PM, Jim Alateras j...@sutoiku.com wrote:

I was wondering whether there were any mechanisms to use ES as a
realtime feed for downstream systems. I have a cluster that gathers
observations from many sensors. I have a need to maintain a list of
realtime counters in REDIS so I want to further process these observation
once they hit the database. Additionally I also want to be able to create
event streams for different type of feeds.

I could do all this outside ES but I was wondering whether there were
mechanisms within ES that will allow me to subscribe to add events for a
particular type or index.

cheers

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9f5b1d11-0be1-461d-a5bd-dd70f1a0b6c1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a3468dc3-2b96-4f00-a921-fba6892b5bba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5