Custom volumeMounts with Elastic Cloud on Kubernetes

[deepflyz (Frank)]

Hi,

I'm trying to install Elastic Cloud on Oracle OCI Kubernetes but I'm facing difficulties to use a custom volumeMounts.

Here is the yml file:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic
  namespace: elastic
spec:
  version: 8.5.3
  nodeSets:
    - name: default
      count: 1
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              volumeMounts:
                - mountPath: /usr/share/elasticsearch/data
                  subPath: elastic-data
                  name: elastic-data-mount
      volumeClaimTemplates:
        - metadata:
            name: elastic-data-mount
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
            storageClassName: "" # That must be used for Provisioning PVCs on the Oracle File Storage Service
            volumeName: elastic-pv

Here is the error message

{
  "@timestamp": "2022-12-13T13:23:25.794Z",
  "log.level": "ERROR",
  "message": "fatal exception while booting Elasticsearch",
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.server",
  "process.thread.name": "main",
  "log.logger": "org.elasticsearch.bootstrap.Elasticsearch",
  "elasticsearch.node.name": "elastic-es-default-0",
  "elasticsearch.cluster.name": "elastic",
  "error.type": "java.lang.IllegalStateException",
  "error.message": "failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?",
  "error.stack_trace": "java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?...."
}

When I mount the PVC, then I can see the created elastic-data folder. But the folder is always empty.

I tried to change the mountPath to /usr/share/elasticsearch. Then a config and a logs folder get created, but no data folder.

Anybody has an idea?
Thanks!

[deepflyz (Frank)]

After a good night sleep and another try, I could make it work by:

1. Create a new storageClass called oci-bv-ext4
2. Create a PersistentVolumeClaim which makes use of the new storageClass
3. Make use of the new storageClass and PVC in the Elasticsearch volumeClaimTemplates

Here is the yml for the storageClass and PVC

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: oci-bv-ext4
provisioner: blockvolume.csi.oraclecloud.com
reclaimPolicy: Retain  # Use this or Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Namespace
metadata:
  name: elastic
  labels:
    name: elastic
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elastic-data-mount-elastic-es-default-0
  namespace: elastic
spec:
  storageClassName: oci-bv-ext4
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi