# Are you using rivers for mission-critical data? Would you? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 26, 2012, 8:43pm UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516 "2012-01-26T20:43:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![derrickburns](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@derrickburns](https://discuss.elastic.co/u/derrickburns)\ **Post date:** [January 26, 2012, 8:43pm UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/1 "2012-01-26T20:43:53Z")
I am considering using a River as the primary data input mechanism for my business. The River will pull data AND persist the data to Amazon S3. Do you know of anyone trusting their mission-critical data to the river? The alternative is to write my own reader that persists the data and then pushes it to S3, outside of an elasticsearch node. I am attracted to the option of putting my code inside a river because of the fail-over mechanism that the river provides. Are you using this feature in production for mission critical data? ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [January 27, 2012, 4:32pm UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/2 "2012-01-27T16:32:59Z")
Our preference has been not to use the rivers. We find using external process more flexible and provides more control. For fail-over, we've implemented our own mechanism, typically designed external processes in active-active configuration for both load balancing and fault tolerance. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Thu, Jan 26, 2012 at 3:43 PM, Derrick [derrickrburns@gmail.com](mailto:derrickrburns@gmail.com) wrote: > I am considering using a River as the primary data input mechanism for my > business. The River will pull data AND persist the data to Amazon S3. Do > you know of anyone trusting their mission-critical data to the river? > > The alternative is to write my own reader that persists the data and then > pushes it to S3, outside of an elasticsearch node. I am attracted to the > option of putting my code inside a river because of the fail-over mechanism > that the river provides. Are you using this feature in production for > mission critical data? ---
**Author:** ![Frederic](https://avatars.discourse-cdn.com/v4/letter/f/4491bb/32.png) [@Frederic](https://discuss.elastic.co/u/Frederic)\ **Post date:** [January 27, 2012, 5:35pm UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/3 "2012-01-27T17:35:45Z")
Same thing here. After posting some questions here [https://groups.google.com/group/elasticsearch/browse\_thread/thread/5ec6aeb8e76a3ad9](https://groups.google.com/group/elasticsearch/browse_thread/thread/5ec6aeb8e76a3ad9) we decided not to leverage rivers. In our case we need to consume a feed of doc changes and we shouldn't lose any new message, so that we have finally implemented our custom job for retrying and logging in case of every possible failure. Of course this ony applies to our specific needs and, at the other hand, we are indexing docs (about 50/sec) one by one, which is less efficient and standard than using bulk indexing with rivers I guess. Cheers, On 27 ene, 13:32, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > Our preference has been not to use the rivers. We find using external > process more flexible and provides more control. For fail-over, we've > implemented our own mechanism, typically designed external processes in > active-active configuration for both load balancing and fault tolerance. > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Thu, Jan 26, 2012 at 3:43 PM, Derrick [derrickrbu...@gmail.com](mailto:derrickrbu...@gmail.com) wrote: > > > I am considering using a River as the primary data input mechanism for my > > business. The River will pull data AND persist the data to Amazon S3. Do > > you know of anyone trusting their mission-critical data to the river? > > > The alternative is to write my own reader that persists the data and then > > pushes it to S3, outside of an elasticsearch node. I am attracted to the > > option of putting my code inside a river because of the fail-over mechanism > > that the river provides. Are you using this feature in production for > > mission critical data? ---
**Author:** ![derrickburns](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@derrickburns](https://discuss.elastic.co/u/derrickburns)\ **Post date:** [January 27, 2012, 7:21pm UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/4 "2012-01-27T19:21:48Z")
Thanks Berkay and Frederic for the info. I think that you have identified a hole in using Rivers for me: what to do with the incoming data should the process fail to consume it, either writing it to S3 or to the index. Can you suggest an specific tools for implementing a simple Active-Passive configuration in Java on AWS? ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [January 28, 2012, 3:49pm UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/5 "2012-01-28T15:49:56Z")
Have not used anything AWS specific. One approach is to have the active instance to send heartbeat to the passive instance as it processes data. If the passive instance does not receive the heartbeat, it would do another check to see whether the active is down, and if it is down, it would take over. It's best to make the decision based on application level checks rather than checking whether server or process is running, etc. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Fri, Jan 27, 2012 at 2:21 PM, Derrick [derrickrburns@gmail.com](mailto:derrickrburns@gmail.com) wrote: > Thanks Berkay and Frederic for the info. I think that you have identified > a hole in using Rivers for me: what to do with the incoming data should the > process fail to consume it, either writing it to S3 or to the index. > > Can you suggest an specific tools for implementing a simple Active-Passive > configuration in Java on AWS? ---
**Author:** ![derrickburns](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@derrickburns](https://discuss.elastic.co/u/derrickburns)\ **Post date:** [January 29, 2012, 4:24am UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/6 "2012-01-29T04:24:33Z")
Thanks Berkay. This makes a lot of sense. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:41am UTC](https://discuss.elastic.co/t/are-you-using-rivers-for-mission-critical-data-would-you/6516/7 "2017-07-06T03:41:09Z")
Closed on July 6, 2017, 3:41am UTC