Plan for upgrading from 2.4 to 5.6?

[ppbrown (Philip P Brown)]

I'm trying to put together a safe plan for upgrading our cluster from v2.4.1 to v5.6

Been doing some googling.
So far, i've read that;

1. "just backing up the data directories is not sufficient", and
2. elastic has built in snapshot and backup capabilities.

HOWEVER: the "snapshot and backup" capabilities are not in-place. They seem to require a separate filesytem.
WHich is going to be horrible.
OUr data size is 15TB.
It will take hours and hours to back that up.
And given how restoring a logical SQL dump generally takes even longer than doing the backup...
I shudder to think how long an elastic restoral of 15TB will take.

So I'm looking for some saner options, that dont involve "build an entirely new cluster to take over from the old one", and also minimize downtime for the cluster, even in the result of failed upgrade.

Currently, all our data is on xfs.
Surely I can do something along the lines of

0. shut down cluster

1. take xfs snapshots

2. also backup.. (what? what else do I need?)

3. do upgrade

4. if things go horribly wrong, revert data to snapshots, and then restore whatever was in step 2.