Reverse River?


(Tim Uckun) #1

I would like to have a river in reverse. Every time a document is inserted
or modified I would like to push that into another destination like a
database. Ideally this would be async or maybe even in batches.

Has anybody done anything like this before?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #2

Some relevant comments:

--
Ivan

On Thu, May 22, 2014 at 8:45 PM, Tim Uckun timuckun@gmail.com wrote:

I would like to have a river in reverse. Every time a document is inserted
or modified I would like to push that into another destination like a
database. Ideally this would be async or maybe even in batches.

Has anybody done anything like this before?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA3K_0Sa0C_4PeXjkr%3DvzuBzhXSYp91wSTs8ThtdOTbKA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #3

Note, if you want to connect a database to ES internals, this is generally
a bad idea. The challenge is performance and transaction safety. ES API can
process many hundreds of thousands of actions in a second in unspecified
order, like a "data hose". No database exist that I know of that can keep
up with such a pace. The price would be a slowdown of ES throughput (async
also means a higher cost of CPU for threads and RAM for the message queue
on the same machine which would compete heavily with ES for resources).
From the viewpoint of designing such an architecture, never do a "reverse
river" for "every time a document is inserted or modified". Instead, just
push the doc to ES and to the database concurrently from your middleware
only if necessary, i.e. after applying a filter condition to your doc. This
would also maintain scalability and transaction safety since there is no
guarantee that ES transport layer does not drop (or repeat) transmissions
at any time.

Beside these design questions, there is a new "mock transport" that has
been added to 1.2.0. Nice side effect is that plugin authors should be able
to completely exchange the transport module or wrap the existing one with a
logging facility. I think about a syslog client for revisable logs that
capture certain ES transport actions and write them into an append-only log
on a central log server, so it's up to the syslog server to keep up with
the pace. With a bit filtering it could be possible to capture certain
documents and log them. If this can work out is still totally unclear, I'm
still in the process of designing :slight_smile:

Jörg

On Fri, May 23, 2014 at 5:45 AM, Tim Uckun timuckun@gmail.com wrote:

I would like to have a river in reverse. Every time a document is inserted
or modified I would like to push that into another destination like a
database. Ideally this would be async or maybe even in batches.

Has anybody done anything like this before?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGbNbxBV20XA0nF_bRQy%3D_g5zZjtLB1eYYm36aSZOsA1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4