Adding volumeAttributesClassName is forbidden

Elastic Stack Elasticsearch
1

After the cluster’s nodeSets are created. I want to specific a volumeAttributesClassName however, the eck operator’s admission webhook is preventing me.

Failed to save resource: (Forbidden) admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "s1" is invalid: spec.nodeSet[2].volumeClaimTemplates: Invalid value: v1.PersistentVolumeClaim{v1.PersistentVolumeClaim{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"elasticsearch-data", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:v1.OwnerReference(nil), Finalizers:string(nil), ManagedFields:v1.ManagedFieldsEntry(nil)}, Spec:v1.PersistentVolumeClaimSpec{AccessModes:v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, Selector:(*v1.LabelSelector)(nil), Resources:v1.VolumeResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList{"storage":resource.Quantity{i:resource.int64Amount{value:1717986918400, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeName:"", StorageClassName:(*string)(0xc000cc54b0), VolumeMode:(*v1.PersistentVolumeMode)(nil), DataSource:(*v1.TypedLocalObjectReference)(nil), DataSourceRef:(*v1.TypedObjectReference)(nil), VolumeAttributesClassName:(*string)(0xc000cc54c0)}, Status:v1.PersistentVolumeClaimStatus{Phase:"", AccessModes:v1.PersistentVolumeAccessMode(nil), Capacity:v1.ResourceList(nil), Conditions:v1.PersistentVolumeClaimCondition(nil), AllocatedResources:v1.ResourceList(nil), AllocatedResourceStatuses:map[v1.ResourceName]v1.ClaimResourceStatus(nil), CurrentVolumeAttributesClassName:(*string)(nil), ModifyVolumeStatus:(*v1.ModifyVolumeStatus)(nil)}}}: volume claim templates can only have their storage requests increased, if the storage class allows volume expansion. Any other change is forbidden

If this isn’t possible, it kind of defeats the purpose of volumeAttributesClass or am I mistake? Any ideas on solving this?

attributes-class.yaml

apiVersion: storage.k8s.io/v1beta1
kind: VolumeAttributesClass
metadata:
  name: s1-data
  labels:
    app.kubernetes.io/name: s1
    app.kubernetes.io/instance: usw2-s1
    app.kubernetes.io/managed-by: Helm
  annotations:
    description: "VolumeAttributesClass for data NodeSet in s1 cluster"
driverName: ebs.csi.aws.com
parameters:
  # Team and service tags
  tagSpecification_1: "Team=my-team"
  tagSpecification_2: "Service=s1"
  tagSpecification_3: "Environment=stg"
  tagSpecification_4: "Cluster=s1"
  tagSpecification_5: "NodeSet=data"    
  # Performance parameters for EBS volumes
  iops: "3000"
  throughput: "125"
  # Volume type and filesystem
  type: "gp3"
  csi.storage.k8s.io/fstype: "ext4"

elasticsearch.yaml

...
volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1000Gi
            storageClassName: es-gp3-high
            volumeAttributesClassName: s1-data

I should mention that, I’ve checked the requirements for this. (eks version, ebs-csi version, feature gate enabled, StorageClass allows for volumeExpansion).

Running

  • ECK Operator 2.16.1
  • Elasticsearch 8.15.3
2

This operator admission validation webhook is even preventing me from switching the name of the VolumeAttributesClassName, which defeats the whole point and suggested use of this new API object.

How do we go about in making a feature change?

error: elasticsearches.elasticsearch.k8s.elastic.co "" could not be patched: admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "" is invalid: spec.nodeSet[3].volumeClaimTemplates: Invalid value: v1.PersistentVolumeClaim{v1.PersistentVolumeClaim{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"elasticsearch-data", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:v1.OwnerReference(nil), Finalizers:string(nil), ManagedFields:v1.ManagedFieldsEntry(nil)}, Spec:v1.PersistentVolumeClaimSpec{AccessModes:v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, Selector:(*v1.LabelSelector)(nil), Resources:v1.VolumeResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList{"storage":resource.Quantity{i:resource.int64Amount{value:107374182400, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Gi", Format:"BinarySI"}}}, VolumeName:"", StorageClassName:(*string)(0xc00274ac60), VolumeMode:(*v1.PersistentVolumeMode)(nil), DataSource:(*v1.TypedLocalObjectReference)(nil), DataSourceRef:(*v1.TypedObjectReference)(nil), VolumeAttributesClassName:(*string)(0xc00274ac70)}, Status:v1.PersistentVolumeClaimStatus{Phase:"", AccessModes:v1.PersistentVolumeAccessMode(nil), Capacity:v1.ResourceList(nil), Conditions:v1.PersistentVolumeClaimCondition(nil), AllocatedResources:v1.ResourceList(nil), AllocatedResourceStatuses:map[v1.ResourceName]v1.ClaimResourceStatus(nil), CurrentVolumeAttributesClassName:(*string)(nil), ModifyVolumeStatus:(*v1.ModifyVolumeStatus)(nil)}}}: volume claim templates can only have their storage requests increased, if the storage class allows volume expansion. Any other change is forbidden

You can run kubectl replace -f /var/folders/4d/rq0c6x8j369fkjpbs5z47y4m0000gq/T/kubectl-edit-3968695677.yaml to try this update again.