CCR - auto-follow problem on data streams

[Wojciech_Kwiecien]

I found this topic Elastic Cross Cluster Replication of Data Stream

But I do not find a solution for me.
I was able to create CCR auto-follow pattern and nothing happened when I ran GET /_ccr/stats
After I follow instructions from Elastic Cross Cluster Replication of Data Stream
and after rollover on source POST traces-apm-default/_rollover something happened.
I received few errors :

{ "auto_follow_stats" : { "number_of_failed_follow_indices" : 134, "number_of_failed_remote_cluster_state_requests" : 0, "number_of_successful_follow_indices" : 0, "recent_auto_follow_errors" : [ { "leader_index" : "traces:.ds-traces-apm-default-2023.12.06-000092", "timestamp" : 1701868421437, "auto_follow_exception" : { "type" : "exception", "reason" : "java.lang.IllegalArgumentException: cannot follow backing index [.ds-traces-apm-default-2023.12.06-000092], because local data stream [traces-apm-default] is no longer marked as replicated", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "cannot follow backing index [.ds-traces-apm-default-2023.12.06-000092], because local data stream [traces-apm-default] is no longer marked as replicated" } } } ]

In another case what I tried to do was to add suffixes but after that I received

"recent_auto_follow_errors" : [
{
"leader_index" : "follower:.ds-traces-apm-default-2023.12.06-000094",
"timestamp" : 1701874886251,
"auto_follow_exception" : {
"type" : "exception",
"reason" : "java.lang.IllegalArgumentException: a backing index name in the local and remote cluster must remain the same",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "a backing index name in the local and remote cluster must remain the same"
}
}
}
],

If someone can take a look and help I will be very grateful.

[Wojciech_Kwiecien]

@adsandie Can you take a look and help

[yago82]

Hi,

The first error says that the local data stream 'traces-apm-default' isn't flagged as replicated anymore. That could happen if the data stream got deleted or changed in a way that messes up the replication process. Check up on its status and make sure it's set up properly for replication.

The second error points out that the names of the indexes in both the local and remote clusters need to match. So, when you're setting up Cross-Cluster Replication (CCR), it's super important that the index names in the main and follower clusters are exactly the same. If you've added a tag or something to the index name in the follower cluster, that might be causing the issue. Just double-check to ensure both clusters have the same index names when you're setting up CCR.

Regards

[Wojciech_Kwiecien]

When I set it up this way and ran GET /traces-apm-default/_ccr/info, nothing was visible in the logs. When I clicked [View Follower Indexes in Index Management], no indexes were created on the target cluster.

I thought that after setting it up, it would take a while to create indexes and start replicating data, but I waited a few hours and nothing happened.

The only action in the visible in logs was when index was rollover on the source cluster.

Is there any option to check the progress of this process?

Do you know how long this process might take? There are 3 indexes in the source cluster, each 50 GB in size

[Wojciech_Kwiecien]

Our version is 7.17.12 for both.
Both cluster have the same settings in traces-apm@custom in component templates

{
  "index": {
    "refresh_interval": "30s",
    "number_of_shards": "1",
    "auto_expand_replicas": "0-0",
    "number_of_replicas": "0"
  }
}

[adsandie (Lawrence)]

There should be no data stream on the destination cluster. If there is, you must delete it.

If you run GET _data_stream/traces-apm-default, you will see "replicated": false. For data stream ccr, you cannot do a snapshot/restore a portion of the data stream and do a ccr.

Another option for you if you do not want to delete the destination data stream is to append a suffix in your auto-follow pattern like "-replicated".