We are preparing to do a rolling upgrade on a fairly large cluster. As part of that, we have done the rolling upgrade on a much smaller development cluster.
We left it half upgraded to see how the services will behave in the middle of the larger upgrade. Our services use Golang olivere elastic library which I don't expect support for that and my issue seems unrelated to it.
The two concerns were the way total hits are returned have changed and the removal of the doc type.
And with the cluster half upgraded (half 6.8 & half 7.9) we do get a slightly different response type depending if the load balancer hits a v6 node or v7. the olivere library naturally breaks if it gets a 7.x response due to total hits not being an integer.
Is this expected and not part of the backwards compatibility that should happen during an upgrade?
https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html implies it should be backwards compatible until the final node is upgraded expect for a few scenarios that can cause that not to happen.
Is it possible to do a rolling upgrade a large cluster (that might take 10hrs to upgrade) from 6.8.6 to 7.9.3 with no downtime or writing code to account for random responses?
It seems it should be the case and that we've done something wrong and I'm not sure what we did wrong in the process. I would just like to make sure I'm not chasing an expected outcome that won't happen because I didn't understand what the docs meant.