Cluster-to-Cluster replication using polling

[dux2]

Hi

I have two DCs (A and P), each of them runs its own ES cluster in Active/Passive deployment,
I would like to mirror all data from A to P continuously with latency as low as possible (up to 60 sec).
I try to avoid using another DB (such as CouchBase) as a primary DB due to the additional computing resources and complexity involved.

I can go ahead using the snapshot and restore mechanism, but I fear that P will not get updated soon enough.
I thought of writing a custom poller for changes that polls A for changes (for example, based on last poll time-stamp) and apply them to P (as suggested by Shay, see reference below).

Is that a viable option? Any drawbacks? Can you suggest another strategy?

References:

Continuous async replication

Correct, this would create new indices. On Apr 21, 2014, at 2:50 PM, Mohit Anchlia mohitanchlia@gmail.com wrote: If I understand this correct with this pattern evey restore is a new index? curl -XPOST "localhost:9200/snapshot/my_backup/snapshot_1/restore" -d '{ "indices": "index_1,index_2", "ignore_unavailable": "true", "include_global_state": false, "rename_pattern": "index(.+)", "rename_replacement": "restored_index$1" }' On Mon, Apr 21, 2014 at 2:32 PM, Andrew Selden andrew.selden@…

Replication Strategies

Curious, if you guys are using ElasticSearch for replication, how do you replicate your data? I'm assuming your main data source is slow, so I was wondering how you go around it to have a faster replication. Do you do your huge/slow query on your main data store and store that in ElasticSearch, or do you guys query expensive tables and store those in ElasticSearch, and then later recombine them? ...or? Do you guys do lazy replication wherein the first time a query is made, …