Cross Cluster Replication Issue

[Prabakar]

We are using Elasticsearch open source 7.13. We are trying to perform Cross Cluster Replication. We followed below steps.

1. Created Index (Leader Index) on Cluster 1.
2. Created cluster 1 as a remote cluster for cluster 2 .
3. We are executing below PUT request to create follower index on cluster 2.

PUT follower_index_Name/_ccr/follow
{
"remote_cluster":"test_remote_cluster",
"leader_index":"leader_index_name"
}

We are getting below error.

{
"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "current license is non-compliant for [ccr]",
"license.expired.feature": "ccr"
}
],
"type": "security_exception",
"reason": "current license is non-compliant for [ccr]",
"license.expired.feature": "ccr"
},
"status": 403
}

Is Cross Cluster Replication available in Open source? or do we need to go with Enterprise edition to achieve CCR?

[warkolm (Mark Walkom)]

Welcome to our community!

@Prabakar

Is Cross Cluster Replication available in Open source?

It is not, no.

CCR is a Platinum and above level feature - Subscriptions | Elastic Stack Products & Support | Elastic

[Prabakar]

Thank you for letting me know. May I know what are all the options available to do the CCR in open source version. Can you share your thoughts on this.

[Christian_Dahlqvist]

There is no way to do that within Elasticsearch with the free basic license. Before CCR was created a reasonably common way to achieve this was to feed the two clusters in parallel, e.g. by having two Logstash instances feed the clusters from Kafka or some other message queue, but whether this is appropriate or not depends on your use case.

[Prabakar]

Ok. Can we follow below approach if we have Active/Passive setup.

1. Take snapshots (For example, Every hour) on cluster 1 and store it on S3 bucket or Google Cloud Storage.
2. Restore it on Cluster 2 from S3 or Google cloud storage bucket.

[dadoonet (David Pilato)]

Yes. That will work as well.

[Prabakar]

Thank you for the response. If it is Active/Active then may I know what are the best ways to achieve CCR apart from feeding the two clusters in parallel?

[Prabakar]

I have enabled platinum 30 day trial. I can able to perform cross cluster replication. But facing an issue while updating data on follower index

Steps:

1. Created Index (Leader Index) on Cluster 1 (For example, Customer)
2. Created cluster 1 as a remote cluster for cluster 2 .
3. Created follower index (customer) on Cluster 2 using remote cluster (cluster 1)
4. Trying to update the record on customer index on cluster 1. It got succeeded and I can able to update the document.
5. Trying to update the record on follower index (customer) on cluster 2. But I am getting below error.

"type": "status_exception",
"reason": "a following engine does not accept operations without an assigned sequence number"

I tried passing _seq_no and _primary_term as well, but no luck

PUT customer/_doc/01?if_seq_no=4&if_primary_term=1
{
"firstName":"Smith",
"last Name": "John",
"seq_no":4,
"_primary_term" : 1
}

Please share your thoughts on this? Is it possible to update the records on follower index after replication? We are looking for Active/Active model. So inserts, updates and deletes on the index will happen on both the clusters. So we are looking for Bi-directional approach.

[Christian_Dahlqvist]

CCR does as far as I know not support active-active replication. Please see this blog post for details. I am not aware of any solution to achieve this in Elasticsearch.

[Prabakar]

Ok. But we have a question on Active/Passive model as well.

1. Cluster 1 we are creating indexes and performing all write operations. Then replication will happen automatically on the other cluster (Cluster 2).
2. If we have major outage on Cluster1 then only Cluster 2 will be available for the operation. As you said Read will work fine, since follower indexes are read only. But what we can do for write here.
   How will we achieve DR (Disaster Recovery) process?

[Christian_Dahlqvist]

I believe you would need to terminate replication in order to transform it into a leader index that you can make updates to and then follow once the other site comes back up.

[Prabakar]

Yes. Replication has been paused by doing select the follower index and pause replication (On cluster 2). When the follower index status changes to Paused, reselect the follower index and choose to unfollow the leader index. Then the index started acting like a normal index to perform insert, update and delete operations. Below is my question.

1. After some point cluster 1 recovered from crash.. Then how the updates or inserts or deletes happened on cluster 2 index will navigate to Cluster 1? Because Cluster 1 already have an index with the same name with old data. How to achieve this?

This is normal Disaster Recovery process. It helps if you share how we can achieve this and it may help us to make the decision to go with Enterprise edition.