Retiring data to another Datacenter

Elastic Stack Elasticsearch
1

Hi,

We are looking for a way to move old indices to a different datacenter, still using the same cluster.

All new data will go to specific "hot" nodes (using shard allocation awareness), and each day an old index will be moved to other "cold" nodes (Again, using shard allocation awareness).

No new data will be indexed in the cold data nodes.

I understand its possible to have split brain situations in this scenario, but i don't mind the cold nodes to be "detached" for some time from the cluster, as long as its not affecting indexing/searching in the "hot" nodes.

When the connection comes back between the cluster they should be able to fully restore the cluster health right?

Will that work?

Thanks

2

You would probably need to restart one "half" of the cluster for this to happen.

Sghort answer, no. Longer answer, don't even try.

3

Thanks for the answer @warkolm.

What if we wanted to completely move our ES cluster to a different datacenter? What would be the best approach to do that?

4

Snapshot and restore.
You can reindex from one into the other too.

5

And what about the live index? the one that constantly have new data being indexed into?

6

Also, what exactly is the reason that it won't work? Can't i use the multiple datacenter solution just during the transfer to the new DC?

I can set the new DC to use a new shard allocation tag, and then move indices one by one. Why would that won't work?

7

Check out Scaling Elasticsearch Across Data Centers With Kafka | Elastic Blog

Are you willing to restart half your cluster to get it to join back? Or to separate it?

8

If its only for the duration of the migration of the data to another datacenter than yeah, i'm willing to restart it. (I believe its not going to fail every few seconds...)

9

You hope, you mean :slight_smile: