Hi,
When rolling through a cluster upgrade is there a recommendation for the order of the upgrade?
Currently, I upgrade masters first. However, the other day we lost a data node which then wouldn't join the upgraded master node for the cluster. This then broke the state management controls of my deployment automation. So, I'm mulling reversing the order and upgrading master nodes last...
You can run into the same sort of situation if one of the master nodes disconnects from the cluster while you're upgrading a different master node. The order is less of an issue than the IMPORTANT section at the bottom of the rolling upgrade docs:
In the unlikely case of a network malfunction during the upgrade process that isolates all remaining old nodes from the cluster, you must take the old nodes offline and upgrade them to enable them to join the cluster.
TLDR: if something goes wrong, you (or your orchestration) is expected to push through with the upgrade to bring the cluster back to health.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.