# How to define multiple path.data pointing to separate volumes or PVC using ECK? **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [February 10, 2020, 4:55pm UTC](https://discuss.elastic.co/t/how-to-define-multiple-path-data-pointing-to-separate-volumes-or-pvc-using-eck/218644 "2020-02-10T16:55:24Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ykozlov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ykozlov/32/62377_2.png) [@ykozlov](https://discuss.elastic.co/u/ykozlov)\ **Post date:** [February 10, 2020, 4:55pm UTC](https://discuss.elastic.co/t/how-to-define-multiple-path-data-pointing-to-separate-volumes-or-pvc-using-eck/218644/1 "2020-02-10T16:55:24Z")
Hi! elasticsearch.yml supports configuration setting where I can specify multiple data path for the ES data node as follows: ```auto path.data: - "/mnt/data1', - "/mnt/data2" - "/mnt/data3" ``` I can define additional volumes and/or PVC using spec.nodeSets.volumeClaimTemplates When I define additional volumes using pod spec.volumeClaimTemplates however, when I specify: ```auto spec: version: 7.5.1 nodeSets: - name: default config: path.data: ["/mnt/data1', "/mnt/data2", "/mnt/data3"] ``` I cant an error from kubernetes operator that path.data is no configurable. Is there a way to achieve this configuration using ECK? Thank you in advance! ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [February 18, 2020, 9:11am UTC](https://discuss.elastic.co/t/how-to-define-multiple-path-data-pointing-to-separate-volumes-or-pvc-using-eck/218644/2 "2020-02-18T09:11:01Z")
Hi @ykozlov! Thanks for opening this thread and sorry for the late answer. I spent some time investigating. Here is an example manifest that should achieve what you want: ```auto apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: multi-volumes spec: version: 7.6.0 nodeSets: - name: default count: 3 config: node.master: true node.data: true node.ingest: true node.store.allow_mmap: false podTemplate: spec: containers: - name: elasticsearch env: - name: path.data value: "/mnt/data,/mnt/data2,/mnt/data3" volumeMounts: - name: elasticsearch-data mountPath: /mnt/data - name: elasticsearch-data2 mountPath: /mnt/data2 - name: elasticsearch-data3 mountPath: /mnt/data3 initContainers: - name: chown-data-volumes command: ["sh", "-c", "chown elasticsearch:elasticsearch /mnt/data && chown elasticsearch:elasticsearch /mnt/data2 && chown elasticsearch:elasticsearch /mnt/data3"] volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard - metadata: name: elasticsearch-data2 spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard - metadata: name: elasticsearch-data3 spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard ``` A few things to note: - See how I override the Elasticsearch settings through an environment variable (`path.data`). That's an undocumented way of providing Elasticsearch settings. I opened an issue in our github repo, because you should be able to also set this through the `config` section: [https://github.com/elastic/cloud-on-k8s/issues/2573](https://github.com/elastic/cloud-on-k8s/issues/2573). - The warning when you use a "blacklisted" setting is just a warning, but does not prevent you from actually using that setting if you know what you are doing. - Make sure one of your volume is named `elasticsearch-data`, otherwise ECK will still create the default `elasticsearch-data` volume for you, in addition to other volumes you ay have defined in the manifest. I created an issue in our github repo so we fix that: [https://github.com/elastic/cloud-on-k8s/issues/2574](https://github.com/elastic/cloud-on-k8s/issues/2574) - See how the `chown-data-volumes` init container is changing permission on the volumes underlying filesystems so the `elasticsearch` user is able to write data into them. This is set up by ECK on the default volume, but not your own volumes. - I created [https://github.com/elastic/cloud-on-k8s/issues/2575](https://github.com/elastic/cloud-on-k8s/issues/2575) so we simplify the overall experience of setting up multiple volumes for Elasticsearch data. ---
**Author:** ![hmz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hmz/32/63440_2.png) [@hmz](https://discuss.elastic.co/u/hmz)\ **Post date:** [February 27, 2020, 1:33pm UTC](https://discuss.elastic.co/t/how-to-define-multiple-path-data-pointing-to-separate-volumes-or-pvc-using-eck/218644/3 "2020-02-27T13:33:33Z")
Hi @sebgl, What if we do not want to use pvc at all? ![Screenshot at Feb 27 16-26-41](https://us1.discourse-cdn.com/elastic/original/3X/f/0/f0ae385ea72c0454bc127fa1b14258cdb53ea46b.png) I don't want to bind persistent volume to master and client nodes, is that possible with ECK? I want to create an architechture like below. ![Screenshot at Feb 27 16-29-51](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78f4202457e419096c8a098896d922d1bae75f02.png) ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [March 2, 2020, 9:23am UTC](https://discuss.elastic.co/t/how-to-define-multiple-path-data-pointing-to-separate-volumes-or-pvc-using-eck/218644/4 "2020-03-02T09:23:35Z")
@hmz see the end of [this doc page](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html) to use emptyDirs for some of your NodeSets (I guess client nodes in your case). Note that if you also use emptyDirs for your master nodes, and happen to loose more than half your master nodes Pods, chances are your cluster won't be able to recover. I would advise against it. It seems fine for client nodes though. On your diagram, please also note that all Pods will talk to each other directly: data Pods won't go through a headless service to talk to Master Pods. It does make sense, as your diagram suggests, to create your own service to route traffic to client Pods only. It does also make sense to scale data Pods independently from masters and clients. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:44am UTC](https://discuss.elastic.co/t/how-to-define-multiple-path-data-pointing-to-separate-volumes-or-pvc-using-eck/218644/5 "2022-11-04T07:44:19Z")