Have a couple of questions regarding the configuration for ES production cluster.
We have a production cluster with 2 data nodes, 3 master eligible nodes and 1 coordinating node. All are configured as dedicated nodes.
- Have read that the recommended setting for replica shards is 2 for production. Since we have only 2 data nodes, does it still make sense to setting the replicas to 2? Since there can be only one replica that can actually be of any use if a data node were to go down.
- If we decide to add one more data node to the cluster later on, will the existing shards and replicas automatically redistribute without down time? And also is there a time delay before the replicas or primary shards become available in the new data node?
- Is it a normal practice to horizontally scale coordinating nodes?
Thanks for you time and help.