ES version: 6.1.0
It is 4 node cluster (2 master, 1 data and 1 client :: Minimum master is set to 2.)
Below is config of both master nodes.
cluster.name: infra_elasticsearch_wos_snapshot_us-west-2_cluster_7
# network settings
network.publish_host:
- _local_
- _ec2:privateDns_
- _ec2:privateIp_
network.bind_host:
- _local_
- _ec2:privateDns_
- _ec2:privateIp_
# http settings
http.max_content_length: 1000mb
http.cors.allow-origin: "/.*/"
http.cors.enabled: true
http.compression: true
# discovery settings
discovery.zen.hosts_provider: ec2
discovery.ec2.groups: infra_elasticsearch_wos_snapshot_us-west-2_cluster_7
discovery.ec2.any_group: false
discovery.zen.minimum_master_nodes: 2
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 3
cloud.node.auto_attributes: true
# allocation settings
cluster.routing.allocation.awareness.attributes: aws_availability_zone
cluster.routing.allocation.awareness.force.aws_availability_zone.values: us-west-2c,us-west-2b,us-west-2a
# roles
node.data: false
node.ingest: false
node.master: true
node.name: i-02d426454203be5bc
# other settings
bootstrap.memory_lock: true
===
Below is error observed in logfile.
[2017-12-22T04:58:46,272][WARN ][o.e.d.z.ZenDiscovery ] [i-02d426454203be5bc] not enough master nodes discovered during pinging (found [[Candidate{node={i-02d426454203be5bc}{iE5NE8PERFiitfE6ryKvdQ}{9FX6CuO5R6ercImJnUylzg}{ip-10-10-200-129.us-west-2.compute.internal}{10.10.200.129:9300}{aws_availability_zone=us-west-2c}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2017-12-22T04:58:49,405][WARN ][o.e.d.z.ZenDiscovery ] [i-02d426454203be5bc] not enough master nodes discovered during pinging (found [[Candidate{node={i-02d426454203be5bc}{iE5NE8PERFiitfE6ryKvdQ}{9FX6CuO5R6ercImJnUylzg}{ip-10-10-200-129.us-west-2.compute.internal}{10.10.200.129:9300}{aws_availability_zone=us-west-2c}, clusterStateVersion=-1}]], but needed [2]), pinging again
Kindly help.