On 11 September 2012 01:28, Jörg Prante email@example.com wrote:
I agree, because I think writing to multiple indices is only a special
case of a more general one.
One exciting use case for a changes stream would be having a client that
is fed by pushed "change events" from one index, and transporting them to
other targets, possibly to a new index locally or even to another index on
a remote cluster, whatever.
A backup/restore operation would be required to complement this use case
by reading the indexed docs and write them in bulk mode to targets, in case
the change stream hook is activated later than the index was created. Not
sure about how the scan/scroll operation works, if it serves the docs in
the order they were indexed it would be perfect. I have to find out how to
handle the condition when the bulk mode should stop and the event mode
This is almost 'transaction log shipping' in a traditional DB model, and I
think would be awesome!
I just wanted to confirm this 'changes stream' is related to Jörg's
Websockets transport plugin? Clint mentioned it as if it was part of the
ES roadmap, and I know Shay talked about an inter-DC replication mechanism
plan at some point, and I just wanted to confirm that the changes stream
concept is what that was about. Maybe this Websockets thing is another way
to think of this?
When I read about the Websockets plugin and the changes stream, it didn't
click about what that meant, but this is quite exciting. I can see how in
our case, and maybe others, one could use the dynamic settings to partition
a current index to live on a certain # nodes, leaving a set of nodes carved
out for the new index (partition the IO load), so that this changes stream
can be interleaved with another process that is doing the raw reindex of
the original data, allowing live changes to continue to be applied, and
allow any VersionConflictException to simply be trapped and ignored as a
case of 'ok well the latest information is now already there'. At the end
of the process you have a newly created index with all recent changes
applied as well, ready for a hot-swap via an alias perhaps.
Is that sort of the idea you had in mind Jörg ?