Horizontal and vertical upgrade of eck cluster

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

As per the documentation, I tried doing upgrades of the cluster using some of the examples given in https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-orchestration.html

a) Horizontal upgrade: I see a new service getting created with every statefulset creation, but under the hood they all belong to one cluster and is accessible via http service 9200 right ? How can I confirm that the shards are being rebalanced after horizontal upgrade ?

❯ kubectl get svc -n elastic-system ✹ ✭
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-exporter ClusterIP 172.20.72.204 9108/TCP 6d1h
elasticsearch-sample-es-data-nodes ClusterIP None 5d20h
elasticsearch-sample-es-data-nodes-2 ClusterIP None 41h
elasticsearch-sample-es-http ClusterIP 172.20.133.254 9200/TCP 5d20h
kibana-sample-kb-http ClusterIP 172.20.94.101 5601/TCP 5d20h

b) Vertical upgrade I tried, but I did specify a wrong RAM size while specifying, and the cluster got into a state where it could not recover/add any new pods after that. I am using 1.0 beta version, and I wonder vertical upgrade is fully supported in this release.

create Pod elasticsearch-sample-es-data-nodes-2-0 in StatefulSet elasticsearch-sample-es-data-nodes-2 failed error: Pod "elasticsearch-sample-es-data-nodes-2-0" is invalid: spec.containers[0].resources.requests: Invalid value: "3Gi": must be less than or equal to memory limit

2

I see a new service getting created with every statefulset creation, but under the hood they all belong to one cluster and is accessible via http service 9200 right ?

Yes. You can ignore the per-StatefulSet service.

How can I confirm that the shards are being rebalanced after horizontal upgrade ?

ECK does not do anything special here. We rely on Elasticsearch to rebalance shards automatically.

Vertical upgrade I tried, but I did specify a wrong RAM size while specifying, and the cluster got into a state where it could not recover

You should normally be able to recover by modifying the Elasticsearch yaml manifest again with a correct RAM size. Note you need to set the proper JVM heap size corresponding to the requested memory.

3

Sorry for delayed reply, thank you so much @sebgl, just editing the elasticsearch object didn't have any issue with vertical scaling. I will figure out a way to update via yaml. Congrats on the GA anouncement for eck 1.0 !