How to achieve CCR for more than 20 indices at once without snapshot

Elastic Stack Elasticsearch
1

Hello,

I have multiple indices already created in cluster A and I want them to be replicated to cluster B , is there a way to create those many follower indices without manual effort as it will increase operational load and snapshot might not help as these indices keep on getting new updates. Kindly request you to help.

Note : if I use kibana console or curl it would take a lot of time to replicate one by one and auto follow pattern was not enabled before. I am new to elastic search

2

This is exactly the problem the cross-cluster replication was designed to solve.

3

Well, the challenge here is there are few indices with unassigned shards during the CCR , I have tried to a reroute which failed , the minimum master node is set to 2 (I have 3 nodes in the CCR cluster , the same in remote as well ). The shards some reason do not get assigned, I have tried everything possible, even ran through multiple solutions you have suggested on other blogs, still does not seem to work. The index size for instance is 40gb ,

1- I have noticed the huge size indices (above 2gb) indices (followers in CCR) are always in pause state and when I try to resume , I get an error like the primary shared of a follower index .. from ccr is unassigned , reason = manual allocation . Ml.machine_mem = .. , xpack.installed = true , transform.node =true , ml.max_open_jobs = 20.) Nested : Indexshardrecoveryexception [failed recovery]

2- when I try reroute with Post like :
Post /_cluster/reroute
Commands: allocate : Index: shard : node: allow primary:
(Specified all values , could not copy paste exact command and mentioned all 3 nodes in node field ) , I get an error : type : named object not found exception resaon : unknown filed allocate. Though the index is in red state

3- tried a full restart of the cluster as well, the cluster.routing allocation enable is also set to all

  1. For some reason the replication happens but few huge size indices are always in pause state in follower stats

Note : the remote's indices are all I green, only in the CCR cluster this happens that I am left with unassigned primaries

4

Hello , requesting your advice on the above

5

It's impossible to offer advice on such a small amount of vague information, sorry. You haven't shared any actual logs or errors or really anything that might help. You haven't even said what version you're using. CCR will definitely normally replicate a 40GB index onto the remote cluster without problems; replicating data onto a remote cluster is exactly what it's designed to do.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.