Ok I'm in a test environment thankfully. Maybe how I upgraded?
I'm updating from 1.5.2 to 1.6.0
Running on Windows 2008 R2
The setup is 4 dedicated data nodes and 1 dedicated master (Plus running extra "sites" client node off master machine). That's all I have as machines.
I updated the data nodes first. Followed the recommended upgrade process. That went ok without a hitch!
So then I shut down the the master updated to 1.6.0 restarted it and boom all data was deleted! It's like it doesn't exist. it's not even on the drives! Gone like disappeared in thin air! 1 billion records down the drain!
Checked my RAID arrays just in case. They seem ok. But highly doubt that I would get 4 arrays dead in one shot lol.
I posted my master and 2 of the nodes logs. There doesn't seem to be anything out of the ordinary except for the master node waiting to get a state back from the nodes.
You will see when the data node came back up it new that 2.2TBs where being used of the .2TBs Now all drives on all nodes are basically empty!