Nodes orchestration with ECK while upgrading to newer versions of Elasticsearch, Kibana etc

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi Community,

I have an experimantal Elastic environment deployed on a Kubernetes cluster and I am working with ECK 2.8.0 and the orchestrator. My stack consists of Elasticsearch 2 master nodes and 2 data nodes and 2 ml nodes nodes, 2 Kibana nodes and 2 Enterprise Search nodes. I have defined the whole cluster in one single YAML file.

Now I want to upgrade all node sets from 8.7.0 to 8.8.0 or later - Elasticsearch, Kibana, Enterprise Search. I have read the orchestration documentation https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-orchestration.html

Does ECK and operator coordinates which components are upgraded first (Elasticsearch->Enterprise Search->Kibana) and also the nodes (node 0, then node 1) in a way that allows a rolling upgrade with no downtime of all my components?

In my experiment wanted to upgrade all three components to 8.8.0 and fired the kubectl apply...

After then all master, data and ml nodes were upgraded at the same time (I think also Enterprise Search) - so there was a downtime.

After reading the documentation my expectation was that ECK/operator first updates the first Elasticsearch master node, then the second, then the first data node, than the second and so forth. And finally the first Enterprise Search, then the second and then the Kibana node by node - because it has dependencies to Elastic and Enterprise S.

If ECK/operator does not automatically coordinates all upgrade steps what is a recommended solution? First change the version for Elasticsearch and so on? Use different YAML files?

And what is the correct order of component upgrades? First Elasticsearch, then Enterprise Search and then Kibana?

Best reards

Sebastian

2

The behaviour your are seeing is because a cluster with 2 master nodes is not considered HA. There is no way to upgrade a two node cluster without downtime. As soon as you upgrade the first master node the cluster loses quorum and you availability.

The ECK operator does not even attempt to do a graceful upgrade with such non-HA clusters but just rolls all Elasticsearch nodes at once.

If you want to have upgrades without downtime you have to move to at least 3 master nodes so that one can be rolled at a time and you have still a working quorum of 2 master nodes to keep your cluster up and running.

The operator will make sure that all Elastic stack applications are upgrade in the right order as long as the relationships between them are defined with the elasticsearchRef, kibaneRef and the other *Ref types.

3

I changed the number of nodes to three, for all my components and it worked! Thanks.

Do I need to have at least 3 nodes for all node types: Elasticsearch master, data and machine learning, Kibana and Enterprise Search?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.