Unfortunately, the testing failed - let me walk through the process.
1 - initially I setup one dedicated master and one dedicate data node using node.roles={master|data} property and also having the cluster.initial_master_nodes set for the one master node.
2 - after step-1, I have a successful cluster formed.
curl http://escluster.local.internal:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.192.106.236 29 32 0 0.04 0.03 0.01 m * ip-10-192-106-236
10.192.104.252 50 33 0 0.01 0.07 0.05 dilrt - ip-10-192-104-252
3 - then I removed this cluster.initial_master_nodes property from elasticearch.yml file of existing nodes and also ensured that new launched systems do not have this property configured.
4 - added one more master node in the pool and cluster was still healthy with one more master in the mix.
curl http://escluster.local.internal:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.192.106.236 29 32 0 0.04 0.03 0.01 m * ip-10-192-106-236
10.192.105.242 19 27 9 0.90 0.60 0.25 m - ip-10-192-105-242
10.192.104.252 50 33 0 0.01 0.07 0.05 dilrt - ip-10-192-104-252
5 - at this stage, ip-10-192-106-236 is still the active master node.
6 - I terminate that node ip-10-192-106-236 assuming that ip-10-192-105-242 will become newly active master node. However, the cluster state went bad again - with following error,
curl http://escluster.local.internal:9200/_cat/nodes?v
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
Any thoughts on this?