# Reverse River? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 23, 2014, 3:45am UTC](https://discuss.elastic.co/t/reverse-river/17685 "2014-05-23T03:45:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Tim\_Uckun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_uckun/32/1256_2.png) [@Tim\_Uckun](https://discuss.elastic.co/u/Tim_Uckun)\ **Post date:** [May 23, 2014, 3:45am UTC](https://discuss.elastic.co/t/reverse-river/17685/1 "2014-05-23T03:45:31Z")
I would like to have a river in reverse. Every time a document is inserted or modified I would like to push that into another destination like a database. Ideally this would be async or maybe even in batches. Has anybody done anything like this before? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [May 23, 2014, 4:54am UTC](https://discuss.elastic.co/t/reverse-river/17685/2 "2014-05-23T04:54:41Z")
Some relevant comments: > > > There should be an integration point for ES and external application where the e…xternal applications should be notified of any document changes or updates that happens in ES. > > CouchDB have a good implementation on it and it would be great if ES can also incorporate something similar or same. > > CouchDB change notification feature - http://guide.couchdb.org/draft/notifications.html -- Ivan On Thu, May 22, 2014 at 8:45 PM, Tim Uckun [timuckun@gmail.com](mailto:timuckun@gmail.com) wrote: > I would like to have a river in reverse. Every time a document is inserted > or modified I would like to push that into another destination like a > database. Ideally this would be async or maybe even in batches. > > Has anybody done anything like this before? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA3K\_0Sa0C\_4PeXjkr%3DvzuBzhXSYp91wSTs8ThtdOTbKA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA3K_0Sa0C_4PeXjkr%3DvzuBzhXSYp91wSTs8ThtdOTbKA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 23, 2014, 7:50am UTC](https://discuss.elastic.co/t/reverse-river/17685/3 "2014-05-23T07:50:02Z")
Note, if you want to connect a database to ES internals, this is generally a bad idea. The challenge is performance and transaction safety. ES API can process many hundreds of thousands of actions in a second in unspecified order, like a "data hose". No database exist that I know of that can keep up with such a pace. The price would be a slowdown of ES throughput (async also means a higher cost of CPU for threads and RAM for the message queue on the same machine which would compete heavily with ES for resources). From the viewpoint of designing such an architecture, never do a "reverse river" for "every time a document is inserted or modified". Instead, just push the doc to ES and to the database concurrently from your middleware only if necessary, i.e. after applying a filter condition to your doc. This would also maintain scalability and transaction safety since there is no guarantee that ES transport layer does not drop (or repeat) transmissions at any time. Beside these design questions, there is a new "mock transport" that has been added to 1.2.0. Nice side effect is that plugin authors should be able to completely exchange the transport module or wrap the existing one with a logging facility. I think about a syslog client for revisable logs that capture certain ES transport actions and write them into an append-only log on a central log server, so it's up to the syslog server to keep up with the pace. With a bit filtering it could be possible to capture certain documents and log them. If this can work out is still totally unclear, I'm still in the process of designing 🙂 Jörg On Fri, May 23, 2014 at 5:45 AM, Tim Uckun [timuckun@gmail.com](mailto:timuckun@gmail.com) wrote: > I would like to have a river in reverse. Every time a document is inserted > or modified I would like to push that into another destination like a > database. Ideally this would be async or maybe even in batches. > > Has anybody done anything like this before? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGbNbxBV20XA0nF\_bRQy%3D\_g5zZjtLB1eYYm36aSZOsA1w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGbNbxBV20XA0nF_bRQy%3D_g5zZjtLB1eYYm36aSZOsA1w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:27am UTC](https://discuss.elastic.co/t/reverse-river/17685/4 "2017-07-06T01:27:27Z")