While performing rolling upgrade from 6.8 to 7.9, the documentation suggest that cluster continue to operate in backward compatibility mode until all nodes are upgraded to newer version. How do we avoid issues with newer version that will only be surfaced once the rolling upgrade is complete(all nodes on newer version) and we are no longer running in backward compatibility mode? At that point we have no option to rollback without a standby hot cluster.
Have you tested the upgrade on a test cluster?
Hi Christian, the intent of the question is to understand the behavior of Elasticsearch before and after the last node is upgraded to higher version during the rolling upgrade. What is the nature of the switch in behavior after the last node upgrade? Usually, If we don't see any problem after most nodes are upgraded, we are reasonably confident and continue on with the upgrade but this assumption breaks if we are only running in backward compatibility mode and there could be surprise break after the last node is updated.
TBC at no point do you really have an option to roll back. As soon as you start the upgrade you are committing to completing it. You must test the upgrade in an isolated environment first!
As long as your 6.x cluster is not logging any deprecation warnings, you shouldn't have any problems when moving to 7.x. The "backward compatibility mode" the docs mention is really just how we describe this constraint:
Nodes running the previous major version will not be allowed to join the fully-updated cluster.