You need to have at least 3 master-eligible nodes on separate hosts. This will allow 2 nodes to reach majority even if one node is unavailable assuming you have configured minimum_mster_nodes correctly. You should therefore make your data nodes also master eligible.
Assuming you have a replica configured for all indices, the cluster will still be able to serve data.
You should monitor disk space and act before it gets full as indices will be made read-only and/or you may suffer from index corruption and data loss.
Elasticsearch will automatically redistribute data across all the data nodes available in the cluster.
In question 1, if we configure replica, the storage will be huge. Suppose we have 5 nodes cluster . In general my cluster size is 50 TB without replica configuration. If we configured the replica the cluster size will be 50 x 5 = 250 TB (Approx) , is this a feasible solution ?
Correct me if I am wrong with the replica concept. Is there any other way to achieve nodes fail-over condition?
In question 2, Suppose we have 90% full storage and we want to add extra storage with out hampering the cluster. Do we need to backup all data directory and increase the storage ?
In question 3, Suppose we have 2 data nodes and in each node path.data size is 15 TB.
In the above scenario, If We add datanode3 , it will be automatically redistribute data and make each data node size is 10 TB??
Correct me if I am wrong.
If your primary shards take up 50TB of storage, configuring 1 replica will double this to 100TB. If you do not have at least 1 replica configured you can not have high availability.
Assuming you have a replica configured, you should be able to take down and modify/upgrade one node at a time while leaving the cluster operable.
Yes, that is basically correct.
If you have 100TB of data you are likely to need more than 2 data nodes. Elasticsearch nodes can not hold an infinite amount of data as the amount of heap available limits this. Exactly how much a node can hold will depend on the use case. Have a look at the following resources: