# "Rivers" & Flume **URL:** \ **Category:** Elasticsearch\ **Created:** [September 21, 2010, 11:02pm UTC](https://discuss.elastic.co/t/rivers-flume/3354 "2010-09-21T23:02:18Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [September 21, 2010, 11:02pm UTC](https://discuss.elastic.co/t/rivers-flume/3354/1 "2010-09-21T23:02:18Z")
I need to drill into the sources and docs about the new River stuff, but I'm very intrigued, I just wanted to sound out whether I'm on the right track here and whether Cloudera's Flume may also be an integration point. From the posts I've read, I can see how rabbitmq et al can be used as a producer/consumer pipeline pushing updates into ES. I'm thinking for a use case we have with a tertiary data centre on the other side of the planet, instead of writing messages, simply write them to a log file, and have Flume reliably deliver these from multiple hosts over to the tertiary and have something like my elasticflume Sink apply them. Is this sort of what Rivers are all about? Source/Sink pattern? Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 21, 2010, 11:42pm UTC](https://discuss.elastic.co/t/rivers-flume/3354/2 "2010-09-21T23:42:26Z")
The idea of "rivers" are elements that are managed within the elasticsearch cluster. There will be only a single river running (per "name") within the cluster, and if the node it runs on fails, then that river will be created on another node. A river make sense when you need to write the "bridging" code yourself that pulls/gets\_pushed data and applies it to elasticsearch. The best example is rabbitmq (or a messaging system), where anyhow you would need to run something that listens for messages from the queue and apply those messages to elasticsearch. In this case, it make sense to run this within the elasticsearch cluster benefitting mainly from the fact that it will only run when the cluster is up, and its support for failover of rivers. Twitter stream is another good example for a river. As for flume, I am not too familiar with it. It looks like its something that you manage yourself (flume instances?), and they basically push data into sinks. With this model, it does not make much sense to have flume as a river. I think a good example that covers both bases and might explain this is couchdb. In couchdb, you can register for a stream of \_changes happening on it. In this case, a couchdb river can be written that would register for changes and apply them. On the other hand, you can write hooks into couchdb that will be called when things change from within couchdb. In this scenario, a river does not really make sense, as simply plugging a hook that calles elasticsearch is a good solution. It feels like flume is applies to the second couchdb case. -shay.banon On Wed, Sep 22, 2010 at 1:02 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > I need to drill into the sources and docs about the new River stuff, but > I'm very intrigued, I just wanted to sound out whether I'm on the right > track here and whether Cloudera's Flume may also be an integration point. > > From the posts I've read, I can see how rabbitmq et al can be used as a > producer/consumer pipeline pushing updates into ES. I'm thinking for a use > case we have with a tertiary data centre on the other side of the planet, > instead of writing messages, simply write them to a log file, and have Flume > reliably deliver these from multiple hosts over to the tertiary and have > something like my elasticflume Sink apply them. Is this sort of what Rivers > are all about? Source/Sink pattern? > > Paul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:19am UTC](https://discuss.elastic.co/t/rivers-flume/3354/3 "2017-07-06T04:19:05Z")