ES migration options

[SpeedDaemon]

I'm currently in the process of moving an Elasticsearch cluster from VM/SAN to physical/local disk.
Current cluster is 6 VMs, each with 2 data nodes and either a master or client node (total 3 master, 3 client, 12 data). Index shards/replicas are 6/1.
New cluster is 4 physicals, each with 7 data nodes, 1 master and 1 client (total 4 master, 4 client, 28 data), same 6/1 shard/replica.
Both clusters will be 7.1.1

When I look around for migration methods, I see things like logstash (did it this way last time, since I wanted to do some processing on the way over), or snapshots, etc.

My question is, can I not just add all the new nodes into the cluster, let it re-organize/stabilize, switch the load balancer to point to the new client nodes, then start removing the old nodes? I never see this offered as a migration solution, so I keep thinking there must be a reason why.

[rugenl (Len Rugen)]

I think in the 7.x world, you need to tell the cluster that the old masters are leaving to keep the master voting working. https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-adding-removing-nodes.html#modules-discovery-removing-nodes

[SpeedDaemon]

I think that's mostly an issue if you shut down a quorum's worth of masters all at once. I'd be shutting down nodes slowly over the space of a week or two in order to give things time to re-replicate and stabilize.

[DavidTurner]

@SpeedDaemon

My question is, can I not just add all the new nodes into the cluster, let it re-organize/stabilize, switch the load balancer to point to the new client nodes, then start removing the old nodes?

Yes, this should work just fine. I recommend fully evacuating each node using a shard allocation filter before shutting it down. It mostly works to just shut them down gradually and wait for green but there are some corner cases that you can avoid with a proper evacuation first. Shut down the old masters one at a time and make sure each one has properly gone from the cluster before shutting down the next one.

@SpeedDaemon

I never see this offered as a migration solution, so I keep thinking there must be a reason why.

I don't know that we document any specific migration techniques do we?

[SpeedDaemon]

Re: evacuating nodes: that was definitely the plan.

I don't think there is any official documentation, but it is a fairly common question according to search results, and I've never seen this offered as a solution.