# Deploying ECK with Network-attached PersistentVolumes (e.g. AWSElasticBlockStore)

**URL:** <https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [September 28, 2021, 12:22pm UTC](https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353 "2021-09-28T12:22:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![brsolomon](https://avatars.discourse-cdn.com/v4/letter/b/8491ac/32.png) [@brsolomon](https://discuss.elastic.co/u/brsolomon)\
**Post date:** [September 28, 2021, 12:22pm UTC](https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353/1 "2021-09-28T12:22:19Z")

</div>

The [ECK](https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html) docs would greatly benefit from an [MCVE](https://stackoverflow.com/help/minimal-reproducible-example) on how to deploy ECK with Network-attached PersistentVolumes and a provisioner such as `kubernetes.io/aws-ebs`. The [Best Practices in AWS](https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-best-practices.html#cloud-aws-best-practices) section suggests using one of Instance Store or EBS-based storage, but the section on [Volume claim templates](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html) does not provide any relevant example.

For example:

* * *

Provision EKS cluster (3 x m5.xlarge workers).

Install ECK custom resource definitions and operator:

```bash
ECK_VERSION=1.8.0

kubectl create \
    -f "https://download.elastic.co/downloads/eck/${ECK_VERSION}/crds.yaml"
kubectl apply \
    -f "https://download.elastic.co/downloads/eck/${ECK_VERSION}/operator.yaml"

kubectl rollout status \
    -n elastic-system \
    --watch --timeout=600s \
    statefulset.apps/elastic-operator

```

Create EBS volume (K8s [docs](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore) hint the volume must be created):

```bash
aws ec2 create-volume \
    --availability-zone=xx-xx-xx \
    --size=64 \
    --volume-type=gp2 \
    --tag-specifications \
        'ResourceType=volume,Tags=[{Key=Name,Value=ElasticData},{Key=Creator,Value=brsolomon}]'
aws ec2 wait volume-available \
    --filters 'Name=tag:Name,Values=ElasticData'

```

Get volume ID:

```bash
sudo yum install -y jq
set -o pipefail
vol_id="$(aws ec2 describe-volumes --filters 'Name=tag:Name,Values=ElasticData' | jq -Mr .Volumes[0].VolumeId)"

```

Create storage class:

```bash
cat <<EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
parameters:
  type: io1
  fsType: ext4
EOF

```

(Attempt to) apply an Elasticsearch cluster specification with custom `volumeClaimTemplates`:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.15.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 8Gi
        storageClassName: aws-ebs
EOF

```

This fails with the `elasticsearch` object stuck in `unknown` state.

What am I doing wrong here? Do I need to pre-create EBS volumes, or can I use dynamic provisioning?

* * *

Info on failure:

```auto
$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
quickstart unknown 7.15.0 ApplyingChanges 6m16s

$ kubectl get pods --selector='elasticsearch.k8s.elastic.co/cluster-name=quickstart'
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 0/1 Pending 0 8m21s
quickstart-es-default-1 0/1 Pending 0 8m21s
quickstart-es-default-2 0/1 Pending 0 8m21s

$ kubectl -n elastic-system logs statefulset.apps/elastic-operator | tail -n25
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.029Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-http"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.039Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-0"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.039Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-1"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.039Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-2"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.325Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.360Z","log.logger":"driver","message":"ES cannot be reached yet, re-queuing","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:34.360Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":186,"namespace":"default","es_name":"quickstart","took":0.342183425}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.361Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":187,"namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.361Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-transport"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.377Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-http"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.387Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-0"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.387Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-1"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.387Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-2"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.671Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.707Z","log.logger":"driver","message":"ES cannot be reached yet, re-queuing","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:44.707Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":187,"namespace":"default","es_name":"quickstart","took":0.346669437}
{"log.level":"info","@timestamp":"2021-09-28T12:43:54.708Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":188,"namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:54.709Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-transport"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:54.727Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-http"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:54.736Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-2"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:54.736Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-0"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:54.736Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-1"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:55.054Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:55.084Z","log.logger":"driver","message":"ES cannot be reached yet, re-queuing","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-09-28T12:43:55.084Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":188,"namespace":"default","es_name":"quickstart","took":0.376270961}

```

From `kubectl describe pods` I notice:

```auto
FailedScheduling
0/3 nodes are available
3 pod has unbound immediate PersistentVolumeClaims.

```

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [September 28, 2021, 12:40pm UTC](https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353/2 "2021-09-28T12:40:02Z")

</div>

Hi Brad,

If you are using EKS and want to use gp2 EBS volumes, you should not have to pre-create the volume manually.  
Rather, you would create the right gp2 storage class, see [Storage classes - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html).  
Then reference that storage class in your Elasticsearch `volumeClaimTemplates `. The corresponding volume will be provisioned for you by EKS on the fly.

---

<div class="post-metadata">

**Author:** ![brsolomon](https://avatars.discourse-cdn.com/v4/letter/b/8491ac/32.png) [@brsolomon](https://discuss.elastic.co/u/brsolomon)\
**Post date:** [September 28, 2021, 12:43pm UTC](https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353/3 "2021-09-28T12:43:37Z")

</div>

Thanks @sebgl, I've updated my original post with an attempt to use an `io1` storage class, but the cluster doesn't start. Does anything look off with that config?

---

<div class="post-metadata">

**Author:** ![brsolomon](https://avatars.discourse-cdn.com/v4/letter/b/8491ac/32.png) [@brsolomon](https://discuss.elastic.co/u/brsolomon)\
**Post date:** [September 28, 2021, 12:53pm UTC](https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353/4 "2021-09-28T12:53:39Z")

</div>

Got this working; I think the issue was with the unnecessary use of `ReadWriteMany` in place of `ReadWriteOnce`.

Updated steps:

```bash
cat <<EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: io1
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
  type: io1
  fsType: ext4
EOF

```

```bash
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.15.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
        storageClassName: io1
EOF

```

```bash
$ kubectl wait --for=condition=ready pod/quickstart-es-default-{0,1,2}
pod/quickstart-es-default-0 condition met
pod/quickstart-es-default-1 condition met
pod/quickstart-es-default-2 condition met

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2021, 12:54pm UTC](https://discuss.elastic.co/t/deploying-eck-with-network-attached-persistentvolumes-e-g-awselasticblockstore/285353/5 "2021-10-26T12:54:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.