Rolling Upgrade Best Practices

When rolling through a cluster upgrade is there a recommendation for the order of the upgrade?

Currently, I upgrade masters first. However, the other day we lost a data node which then wouldn't join the upgraded master node for the cluster. This then broke the state management controls of my deployment automation. So, I'm mulling reversing the order and upgrading master nodes last...


You can run into the same sort of situation if one of the master nodes disconnects from the cluster while you're upgrading a different master node. The order is less of an issue than the IMPORTANT section at the bottom of the rolling upgrade docs:

In the unlikely case of a network malfunction during the upgrade process that isolates all remaining old nodes from the cluster, you must take the old nodes offline and upgrade them to enable them to join the cluster.

TLDR: if something goes wrong, you (or your orchestration) is expected to push through with the upgrade to bring the cluster back to health.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.