How to increase elastic pod volume

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

My cluster is in yellow state as it does not have enough space to allow shard. I initially tried to increase EBS volume size, but elastic node does not seem to recognize the added storage by querying /_cat/allocation?v.

Next I tried to upgrade the pod's volumeClaimTemplates
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 120Gi
storageClassName: gp2enc

but admin webhook rejected such a request. See error information below. How can I increase the pod template volume to get my cluster out of yellow state?

Error: UPGRADE FAILED: cannot patch "quickstart" with kind Elasticsearch: admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "quickstart" is invalid: spec.nodeSet[0].volumeClaimTemplates: Invalid value: v1.PersistentVolumeClaim{v1.PersistentVolumeClaim{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"elasticsearch-data", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:v1.OwnerReference(nil), Finalizers:string(nil), ClusterName:"", ManagedFields:v1.ManagedFieldsEntry(nil)}, Spec:v1.PersistentVolumeClaimSpec{AccessModes:v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, Selector:(*v1.LabelSelector)(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList{"storage":resource.Quantity{i:resource.int64Amount{value:128849018880, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeName:"", StorageClassName:(*string)(0xc000644ff0), VolumeMode:(*v1.PersistentVolumeMode)(nil), DataSource:(*v1.TypedLocalObjectReference)(nil)}, Status:v1.PersistentVolumeClaimStatus{Phase:"", AccessModes:v1.PersistentVolumeAccessMode(nil), Capacity:v1.ResourceList(nil), Conditions:v1.PersistentVolumeClaimCondition(nil)}}}: Volume claim templates cannot be modified

2

VolumeClaimTemplates are immutable and that is a limitation of StatefulSets in Kubernetes.

The workaround is to add another NodeSet with a different name to your Elasticsearch manifest with larger volumes and optionally remove the old NodeSet. This means unfortunately that data has to be copied around between nodes, but as said before this is a limitation of StatefulSets in Kubernetes.

See https://github.com/elastic/cloud-on-k8s/issues/325 for more details on the workaround and the related discusssion.