This is the old datacenter awareness issue again. I've done a thorough search and haven't found any good solution to moving an ES cluster to a new DC, so I'm hoping some one here has a viable method!
So we're moving datacenters and I've the new ES cluster up at the new DC (let's call it DC2). Both the old (DC1) and new (DC2) are running the same ES version (0.90.13), but have a different number of datanodes, as well as DC1 datanodes are on CentOS and DC2 datanodes are Windows 2012R2. Yes, I know, none of this is ideal, but it's what I've been given, so...
I've been testing tools like ElasticDump and Knapsack, and they're fine for our smaller indices, but the one critical index is 100G, and neither tool would be able to move that data within the outage window, not to mention we don't even have the bandwidth between the DCs to make that work.
So I've a couple of options I'm working on:
- Use cluster awareness
I've been playing with the cluster awareness settings, but I'm not sure it's going to get me what I want. Ideally, I'd be able to add the new DC2 nodes to the DC1 cluster, and set it so that the DC2 nodes only ever get replicas, they should never have primary shards. At the same time, I do not want to have less than a replica factor of 1 on the DC1 cluster as it's still our production system and I cannot reduce the redundancy of it.
I haven't been able to come up with a set of awareness settings that would result in this scenario, or perhaps the correct sequence to do so. Is this even possible to do via the 0.90.13 settings?
(And someone really needs to rewrite the section of the manual on forced awareness, I've read it a dozen times and it still doesn't make sense to me. )
- rsync shards to new DC
This is complicated by the DC2 datanodes being W2012R2, but let's say I can get my hands on a CentOS node in DC2. If I rsync the shards over the next coming weeks continuously, my delta during the outage should be small enough to move quickly. Can I simply start that node up cold and have it recognize the rsync'ed shards, or would I have to some how notify it that it has shards locally?