I am currently working on building out an Elasticsearch cluster. I have sent out for the hardware that I'm planning to use and created another topic with the architecture that I'm planning and hardware specs to see if there were any suggestions. One thing that came up was the use of Docker to run multiple Elasticsearch instances on my servers because I have 128Gb of RAM on each (cuz ram is cheap) and it is not suggested that you go over a 32GGB heap size. I have figured out how to make this all work in a virtual environment but my current concern is shard allocation.
I currently have planed to use 2 shards with 1 replica. I planed for this because I have 4 hot servers and with this allocation I would lose 2 servers worth of storage space in effect due to the replica shards but I can also completely lose 2 hosts without any actual data loss. This is important because the hot nodes are all NVME storage so no RAID. They are SSD's so more reliable but a single disk failure is going to completely destroy any data on that host. My concern is that if I set up each data node to run say 3 docker containers with a 32GB heap each that some of the indexes may end up storing a primary shard and its replica on 2 containers on the same host. A failure of that host in that case would cause unrecoverable data loss for those indexes.
My question is if there is a way to prevent that from happening at all? Or is ES just smart enough not to do something like that by understanding that it is in a container on host1 so do not store replicas on containers in that same host? I would rather not start increasing the number of replicas because that is going to further decrease the amount of logs that I can store on the hot storage hosts and since we have never tried centralizing our logs, we are just making a wild guess at how much storage we need. I may make some edits to the number of shards or replicas later once we have a better idea of how much of this storage we are going to use up.