Avoiding huge data replication in case of failure

(dexlesshermi) #1

I'm using elastic in a production environment, and I thought of the following scenario:
Let's say I have N servers that form a single cluster. I have N-1 replicas defined (overall each servers holds N-1 replica shards for every primary shard).
Each server has a group of disks defined in RAID 0 for improved performance, and the elastic get's that RAID 0 as a single block device.
If one of the disks in the i-th server is corrupted, I can basically throw the whole disk, since nothing can guarantee the integrity on that RAID 0 in the i-th server's storage anymore. Then, I would probably shut down the server, install a new disk on that server, and let elastic synchronize the server with the new disk into the cluster.
Assuming that the corrupted disk held M Terabyte of data, that synchronization would cost me a huge amount of unwanted network load.
Hence, I would expect the immediate solution would be to create some fail tolerant RAID in every server (e.g. RAID 50), but for some reason, I saw an explicit recommendation in the elastic docs saying:
"Using RAID 0 is an effective way to increase disk speed, for both spinning disks and SSD. There is no need to use mirroring or parity variants of RAID, since high availability is built into Elasticsearch via replicas."

(Igor Motov) #2

You can also mount your disks separately (instead of putting them all into a RAID) and use multiple data path so elasticsearch can allocated different shards on different disks. Depending on your load, it might not be as efficient and fast as RAID 0, but in this case only shards that were on the disk that died and shards that you were actively indexing into will need to be copied in case of the failure.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.