How to set path.data set to multiple paths within docker

[Petr.Simik]

I found it is possible to configure path.data to multiple paths
https://www.elastic.co/guide/en/elasticsearch/reference/master/path-settings.html

I want to add ssd1+ssd2 as a disk to elastic-node1 and ssd3+4 to elastic-node2
I also want to add directory from non-ssd /mnt/data for snapshots and warm tier indexes..

user@host3:~$ df -h
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdb xfs 26T 34M 26T 1% /mnt/data
/dev/mapper/ssd1-volume xfs 745G 33M 745G 1% /mnt/ssd1
/dev/mapper/ssd2-volume xfs 745G 33M 745G 1% /mnt/ssd2
/dev/mapper/ssd3-volume xfs 745G 33M 745G 1% /mnt/ssd3
/dev/mapper/ssd4-volume xfs 745G 33M 745G 1% /mnt/ssd4

how can I achieve this configuration in docker-compose ?
How can I add more volumes to elastic node using docker config?

thank u

user@host3:~/es-node31$ cat docker-compose.yml
version: '2.2'
services:
es-node31:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
container_name: es-node31
environment:
- node.name=es-node31
- "discovery.zen.ping.unicast.hosts=host2.domain.int,host3.domain.int"
- cluster.name=cem-docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms24g -Xmx24g"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
- 9300:9300
cpuset: "0-11"
network_mode: host
volumes:
- /mnt/ssd1/esdata1:/usr/share/elasticsearch/data
- /mnt/ssd2/esdata2:/usr/share/elasticsearch/data ??? how to set this ???

[Petr.Simik]

I am thinking of the approach to define elasticsearch.yml outside docker and define it

-v custom_elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
...
path:
data:
- /usr/share/elasticsearch/data1
- /usr/share/elasticsearch/data1
...

and in docker compose file link it this way

volumes:

• /mnt/ssd1/esdata1:/usr/share/elasticsearch/data1
• /mnt/ssd2/esdata2:/usr/share/elasticsearch/data2

or

create and build docker image with the same setup

FROM docker.elastic.co/elasticsearch/elasticsearch:7.4.0
COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/

the first approach seems better/simplier
any experiences appretiated?

[DavidTurner]

Both sound fine. Another option (that I prefer) is to combine your pairs of disks together (using e.g. RAID0 or LVM) and offer a single filesystem to each node, which fits more nicely with the default docker-compose setup that expects a single data path.

[Petr.Simik]

Finally my docker file looks like:
docker-compose v2 has cpuset parameter to limit specific CPU, but docker-compose v3 support this only in as part of resource,deploy available only in swarm or docker stack.
The workaround is to use docker-compose --compatibility up -d

the storage i realised to use lvm to create single volume group rather than attaching 2 disk volumes.

the previous option wouldn't work because there are no 2 data directories inside default docker image so we would have to created them before build. Using 1 disk via LVM was easier.

I am not 100% sure if this is good config for production. I do not like --compatibility options and I concern if this would cause some problems in future.

docker-compose.yml

version: '3.3'
services:
es-oscar21:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
container_name: es-oscar21
restart: always
environment:
- node.name=es-oscar21
- "discovery.zen.ping.unicast.hosts=oscar3.corp,oscar2.corp"
- http.cors.enabled=true
- http.cors.allow-origin=*
- cluster.name=corp-docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms24g -Xmx24g"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /mnt/ssd1/esdata1:/usr/share/elasticsearch/data
- /mnt/ssd1/eslog1:/var/log/elasticsearch
ports:
- 9200:9200
- 9300:9300
deploy:
resources:
limits:
cpus: '12'
network_mode: host
es-oscar22:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
restart: always
container_name: es-oscar22
environment:
- node.name=es-oscar22
- "discovery.zen.ping.unicast.hosts=oscar3.corp,oscar2.corp"
- http.cors.enabled=true
- http.cors.allow-origin=*
- cluster.name=corp-docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms24g -Xmx24g"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /mnt/ssd2/esdata2:/usr/share/elasticsearch/data
- /mnt/ssd2/eslog2:/var/log/elasticsearch
ports:
- 9201:9200
- 9301:9300
deploy:
resources:
limits:
cpus: '12'
network_mode: host