Migration data from a cluster to another cluster

Elastic Stack Elasticsearch
1

Hello, dear community,

Assume that I have got 2 different clusters(Cluster-A and Cluster-B). Each cluster has 3 master nodes, 2 hot nodes, 2 warm nodes, and 2 cold nodes.
My goal is to migrate all data from Cluster-A to Cluster-B. I know that everybody will suggest I take the last snapshot in Cluster-A and move it to the proper backup path in Cluster-B then restore backup on Cluster-B.

If I apply this way, could I move all data from Cluster-A to ClusterB?

Step1; Add all nodes(9 nodes) in ClusterB into Cluster-A. After being added all nodes, shard allocation will be starting from Cluster-A nodes to Cluster-B nodes.

Step2; Wait until complete the shard allocations. After completion of shard allocation, shutdown the Cluster-A

I am looking forward to your answer. thanks

2

I don't think this will work, once a node has joined a cluster it won't join a different cluster.

So your data nodes that joined Cluster B won't be able to join Cluster A.

To migrate data between cluster you have a few options, you can use the Snapshot and Restore API, use the Reindex API to reindex from remote, or use Logstash with the Elasticsearch input pointing from one cluster and the Elasticsearch output pointing to another one, which is kind similar as the reindex from remote.

3

If you want to add the new nodes to the existing cluster, setup allocation rules to move shards to them, then shutdown the old nodes, it'll work. You will still technically have the same cluster though, as you've essentially expanded the original cluster then shrunk it.

If that's what you are implying, and not literally starting a new cluster and adding it to the existing one, then go for it.

4

hi @warkolm,
Actually, I mean the new cluster(Cluster-B). Not just data nodes will be added. I will also add the master nodes belonging the Cluster-B. Because Cluster-B also has 3 master nodes like an old cluster(Cluster-A). That is new cluster will have 6 master nodes, 4 hots, 4 warms, and 4 cold nodes. After completing the allocation, I will shut down Cluster-A. will it work?

5

You cannot join clusters. So if you are literally talking about taking these new nodes and creating a cluster called cluster-b, you cannot take those nodes and join them to cluster-a.

If you're referring to cluster-b as a concept that doesn't contain the original nodes, and you want the new nodes to be the only ones in the cluster, the that'll work. But it'll still technically be cluster-a from an Elasticsearch point.

Otherwise you need to be clear on your terminology here please.

6

@warkolm,
Cluster-A has 9 nodes(3 masters, 2 hot, 2 warms, 2 colds). Cluster-B is the same as well... All data is stored in Cluster-A. I want to move these data to the Cluster-B via the allocation method.
cluster name of Cluster-A is "hello-Elasticsearch-A"

  • I will make the cluster name of Cluster-B to "hello-Elasticsearch-A" to be able to add to Cluster-A as a member by updating the Elasticsearch.yml file

  • I will wait to complete the shard allocation from Cluster-A to Cluster-B(new member of Cluster-A)

  • After completing the shard allocation, I will shut down the Cluster-A nodes(9 nodes) and Cluster-B will stay as a "hello-Elasticsearch-A"

is that clear?

7

Again, that won't work.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.