PV not attaching to Elasticsearch pod

Purushotham_Reddy · February 1, 2022, 8:44am

Hi All,

I am facing an issue i.e PV is not getting attached/mounted to Elasticsearch pod.

Background:

I have setup Elasticsearch cluster using eck operator. Details are as follows. Cluster is setup in OKE(Oracle kubernetes engine).

ES Version: 6.8.21
Master Nodes: 3
Data nodes 9

Each data node Volume is 500GB.

Everynow and then we are facing issues with storage. Hence I decided to update the storage from 500GB to 750GB.

So I updated the size of PVC of data-0 which updated its respective PV. Then I restarted the data-0 pod however the pod is not going into running state. Then I had manually updated the Block Volume from 500GB to 750 GB and then tried. Same error. Error is follows.

Error

MountVolume.MountDevice failed while expanding volume for volume "<ocid id>" : MountVolume.NodeExpandVolume failed : expandfs command failed, status: Failure, reason: Invalid command; got [/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/oci expandfs {"kubernetes.io/fsType":"ext4","kubernetes.io/pvOrVolumeName":"<ocid id>","kubernetes.io/readwrite":"rw"} /dev/disk/by-path/<redacted> /var/lib/kubelet/plugins/kubernetes.io/flexvolume/oracle/oci/mounts/<ocid-id>]

<ocid-id> -> Oracle cloud identifier similar to ARN in Amazon.

I know this is something related OCI stuff. I just want to know if anyone run into this issue before. Any guidance or help on this?

Purushotham_Reddy · February 15, 2022, 5:55am

Hi Elastic team,

Could you please provide your suggestions on this?

I think this init container "elastic-internal-init-filesystem" is trying to prepare the file system and not able to pick up the changes done manually(PVC, PV, Underlying block volume). even after replacing this init container command content with something else(i.e echo "hello") it is not working. Kindly do the needful.

Michael_Montgomery · February 22, 2022, 3:13pm

github.com/oracle/oci-cloud-controller-manager

Failed to resize PVC - StorageClass "oci-bv"

opened 06:44AM - 16 Jun 21 UTC

zhihao-infstones

feature

## Is this a BUG REPORT or FEATURE REQUEST? Choose one: BUG REPORT or FEATURE… REQUEST  ## Versions **CCM Version**: v 0.9 **Environment**: - **Kubernetes version** (use `kubectl version`): ``` ubeadm version kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:28:32Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"} ``` - **OS** (e.g. from /etc/os-release): ``` DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" ``` - **Kernel** (e.g. `uname -a`): ``` Linux inst-tnknj-dev-kube-instgrp-master-infpools-org 5.4.0-1035-oracle #38~18.04.1-Ubuntu SMP Wed Jan 6 21:03:06 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ``` - **Others**: ## What happened? I have installed the OCI CSI plugin ,and set the allowVolumeExpansion: true on the StorageClass, but it cannot resize the pvc ``` kubectl describe sc oci-bv Name: oci-bv IsDefaultClass: No Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"oci-bv"},"provisioner":"blockvolume.csi.oraclecloud.com","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"} Provisioner: blockvolume.csi.oraclecloud.com Parameters: <none> AllowVolumeExpansion: True MountOptions: <none> ReclaimPolicy: Delete VolumeBindingMode: WaitForFirstConsumer Events: <none> ``` ## What you expected to happen? We are expecting that the pvc can be resized with "oci-bv" with AllowVolumeExpansion: True ## How to reproduce it (as minimally and precisely as possible)? 1. Set the allowVolumeExpansion: true on the storageClass ``` allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"oci-bv"},"provisioner":"blockvolume.csi.oraclecloud.com","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"} creationTimestamp: "2021-06-16T05:39:38Z" name: oci-bv resourceVersion: "4461" selfLink: /apis/storage.k8s.io/v1/storageclasses/oci-bv uid: 9a257ec6-c8d1-428e-9797-c2a7fb3f350a provisioner: blockvolume.csi.oraclecloud.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE oci-bv blockvolume.csi.oraclecloud.com Delete WaitForFirstConsumer true 60m ``` 2. Request a change in volume capacity by editing the PersistentVolumeClaim's spec.resources.requests.storage field. For example, change the following PVC from having a 50 gibibyte (GiB) disk to having a 104 GiB disk. ``` # pvc-demo.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-demo spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi ``` After editing: ``` # pvc-demo.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-demo spec: accessModes: - ReadWriteOnce resources: requests: storage: 104Gi ``` 3. After that, should see the new volume in the status.capacity field get updated, but it still show the same ```spec: accessModes: - ReadWriteOnce resources: requests: storage: 104Gi storageClassName: oci-bv volumeMode: Filesystem volumeName: csi-ef97ef8e-1a3a-4b47-95e1-b17d42a0a183 status: accessModes: - ReadWriteOnce capacity: storage: 50Gi phase: Bound ``` 4.The storage capacity is still 50Gi, did not get changed ## Anything else we need to know?

From what I'm reading, volume expansion isn't supported within Oracle's CSI.

system · March 22, 2022, 3:13pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.