Cross Cluster Replication - Active/Active?

Elastic Stack Elasticsearch
1

Hello. We need to create two Elasticsearch clusters, with each cluster in a different date centre. The data in the clusters will be accessed by users in an active/active manner, with traffic being balanced across the data centres. The primary reason for this approach is high-availability.

I understand that Cross Cluster Replication is on the road map.

Can you tell me:

  1. If this will support an active/active configuration or just active/warm stand-by?

  2. At what time this year will Cross Cluster Replication functionality be available?

Thanks
Howard

2

It will not, it will support one-way active replication at the index level. We do not think this limits your use case because you can set up two indices with one active in one cluster and one active in the other cluster and have these each one-way replicate to the other cluster. Then, searches can be done over both indices.

I would be interested to hear if you think this does not solve your use-case, and any feedback that you have in general.

We do not provide any indication of release dates.

3

Hello Jason. Thanks very much for your prompt reply. Things are clearer and I think I can simplify further...

Really we just want the same index data to be accessible in two different data centres, with H.A.

Is this a good approach...We take an approach that there is a one "master" DC/Cluster index on which read and write operations are performed. xCDR is used to push this to the other, read-only, DC/Cluster index?

My concern would be that the indices could be out of sync, at least for some amount of time.

We could mitigate this by using session affinity at the network level but does xCDR have functionality to deal with this e.g. does it have immediate push updates (or is it all run on a schedule)?

Thanks again

4

There will be some lag. I can not quantify at this time, and it's dependent on your infrastructure anyway, but our aim would be with high-throughput reliable connections between the clusters that the lag would be on the order of seconds, not minutes. We are still in early stages on development of this feature, so I can only give indications of our goals here.

CCR (we call it cross-cluster replication instead of XDCR = cross-datacenter replication) will operate on a pull model instead of a push model. A following index will pull from a leader index. We think this is the right model because it means that a leader index does not have to maintain state about its followers (instead, the followers maintain state about their leader). The leader maintains a sequence of operations and a marker below which it is is safe for any followers to pull. The follower pulls these operations as soon as it observes the marker advance; by default, the follower polls this marker every 500ms. Note: this is subject to change, this is only guidance on our current implementation and thinking.

Does that help?

5

Hi Jason, that's very helpful - thanks. I will keep an eye out for CCR release data announcements.

BTW, is there a beta channel which would allow us to have early viability of the CCR functionality?

Thanks again.

6

At this time there is not, we are still too early stages for anything like that.

Thanks for your interest.

7

Cool; cheers.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

9

As of Elasticsearch 6.7.0, cross-cluster replication (what we call cross-datacenter replication) is available and production-ready in Elasticsearch. Check out our announcement blog post.

2 Likes