# Discovery with ECS, Dynamic Initial Master Nodes

**URL:** <https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149>\
**Category:** Elasticsearch\
**Created:** [May 28, 2019, 4:50pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149 "2019-05-28T16:50:52Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 4:50pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/1 "2019-05-28T16:50:52Z")

</div>

I am trying to setup a cluster with Elastic 7 on Amazon ECS. But I would like to get a bit of understanding on an approach for setting up the initial master nodes. I can bring up my stack if i explicitely assign each container host in my elasticsearch.yml. However, I dont want to create a new config file every time the underlying architecture changes. Is there a way for the EC2 Discovery plugin to find the initial master data nodes without assigning them explicitely?  
This works:

```auto
cluster.name: test
network.host: 0.0.0.0
network.publish_host: _ec2:privateIp_
transport.publish_host: _ec2:privateIp_
discovery.seed_hosts: {{COMMA SEPERATED IPS}}
cluster.initial_master_nodes:
  - LIST OF IPS
node.name: _ec2:privateIp_
discovery:
  zen:
    hosts_provider: ec2
    minimum_master_nodes: 3
  ec2:
    host_type: private_ip
    tag.Name: tag
    proxy.host: proxy_url
    proxy.port: port
    protocol: http
cloud.node.auto_attributes: true
action.destructive_requires_name: true
bootstrap.memory_lock: true
xpack.security.enabled: false
xpack.ml.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false

```

But again, I dont want to explicitely note the IP's. Would implementing a Coordinating node fix this? Any ideas?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:13pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/2 "2019-05-28T17:13:32Z")

</div>

> [@brhardwick](#):
>
> However, I dont want to create a new config file every time the underlying architecture changes.

Once your cluster has formed, the `cluster.initial_master_nodes` setting is ignored and can be removed. Starting up a stateful service like Elasticsearch for the first time is always a bit special, but once it's up and running you can adjust the architecture fairly freely.

If you are using the EC2 seed provider then you do not need to set `discovery.seed_hosts` at all.

`discovery.zen.minimum_master_nodes` is ignored and deprecated in version 7. You should remove this.

`discovery.zen.hosts_provider` is renamed to `discovery.seed_providers` in version 7, although the old name still works for now. You should adjust your config to use the new setting name.

---

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 5:17pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/3 "2019-05-28T17:17:09Z")

</div>

I understand that the setting is ignored after startup, but I cant get the cluster to find master data nodes without it. This is the key problem. I want my cluster to dynamically find the master nodes without me setting them explicitly. If I am using the EC2 Discovery plugin, should that setup the initial master nodes for me?

Can you explain " If you are using the EC2 seed provider then you do not need to set `discovery.seed_hosts` at all."? Are you referring to the EC2 Discovery plugin?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:22pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/4 "2019-05-28T17:22:20Z")

</div>

> [@brhardwick](#):
>
> If I am using the EC2 Discovery plugin, should that setup the initial master nodes for me?

No, the initial master nodes are not really anything to do with discovery. They're there to hold the first election in a brand-new cluster.

> [@brhardwick](#):
>
> Are you referring to the EC2 Discovery plugin?

Yes, sorry, I did indeed mean the `discovery-ec2` plugin.

---

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 5:29pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/5 "2019-05-28T17:29:13Z")

</div>

> [@DavidTurner](#):
>
> No, the initial master nodes are not really anything to do with discovery. They're there to hold the first election in a brand-new cluster.

Then what you are saying is there is no way to dynamically create the first election in a brand new cluster? I have to hard-code that?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:35pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/6 "2019-05-28T17:35:27Z")

</div>

> [@brhardwick](#):
>
> Then what you are saying is there is no way to dynamically create the first election in a brand new cluster? I have to hard-code that?

I'm not sure I'm following. Are you creating brand-new clusters on a regular basis? How are you orchestrating this? How are you doing the other one-off configuration tasks needed on new clusters, like installing templates and setting up snapshot repositories?

Could you, for instance, give the master nodes more predictable names?

---

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 5:39pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/7 "2019-05-28T17:39:47Z")

</div>

We have a use case where we will need to bring down and startup a brand new cluster and that is what im trying to orchestrate.

For one off configurations, we wait until the stack has been created and the automatically curl the API to update settings in our CloudFormation scripts.

I wouldnt want to give each node a different hard-coded name as this would require a different docker file for each node. But, I could create a coordinating node and have a hard-coded name for _that_ which i could reference in my master/data nodes. I think you refer to this strategy elsewhere.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:48pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/8 "2019-05-28T17:48:07Z")

</div>

Ok, I'm not too familiar with CloudFormation so maybe someone else can comment on other possibilities too.

> [@brhardwick](#):
>
> I wouldnt want to give each node a different hard-coded name as this would require a different docker file for each node.

This doesn't seem so bad to me. It'd just be for the master-eligible nodes, not all nodes, and there's normally only three of them. Can you explain in more detail why you wouldn't want to do this?

> [@brhardwick](#):
>
> I could create a coordinating node and have a hard-coded name for _that_ which i could reference in my master/data nodes.

Yes, another way to hold the first election is to have one special node that elects itself and then all the other nodes just join it. This isn't robust to the case where this one special node doesn't start successfully, but maybe it's easier for you to orchestrate.

Note that it's just the master-eligible nodes that matter for the first election. Master-ineligible nodes (i.e. data nodes, assuming you have dedicated master nodes) completely ignore the `cluster.initial_master_nodes` setting.

---

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 5:52pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/9 "2019-05-28T17:52:50Z")

</div>

> [@DavidTurner](#):
>
> This doesn't seem so bad to me. It'd just be for the master-eligible nodes, not all nodes, and there's normally only three of them. Can you explain in more detail why you wouldn't want to do this?

Three is still too many as ECS would then have to dynamically choose which docker image to use, and we would have to maintain 3 different almost identical images. This, i dont think is a good idea.

> [@DavidTurner](#):
>
> Yes, another way to hold the first election is to have one special node that elects itself and then all the other nodes just join it.

_This_ seems to be a good approach. Would the config look something like this?

```auto
cluster.name: test
node.data: false
node.master: true
network.host: 0.0.0.0
network.publish_host: _ec2:privateIp_
transport.publish_host: _ec2:privateIp_
cluster.initial_master_nodes:
  - election_node
node.name: election_node
discovery:
  zen:
    hosts_provider: ec2
  ec2:
    host_type: private_ip
    tag.Name: somethinghere
    proxy.host: somerthinghere
    proxy.port: somerthing
    protocol: http
cloud.node.auto_attributes: true
action.destructive_requires_name: true
bootstrap.memory_lock: true
xpack.security.enabled: false
xpack.ml.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:58pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/10 "2019-05-28T17:58:52Z")

</div>

> [@brhardwick](#):
>
> Would the config look something like this?

Yes, that looks about right (except `discovery.zen.hosts_provider -> discovery.seed_providers`).

Note that you wouldn't set `cluster.initial_master_nodes` at all on any other nodes in this case.

You might also like to call [this API](https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-adding-removing-nodes.html#modules-discovery-removing-nodes) before shutting the election node down:

```auto
POST /_cluster/voting_config_exclusions/election_node

```

and then this after it's gone:

```auto
DELETE /_cluster/voting_config_exclusions

```

That'll make sure that the cluster is ready to lose this node before it goes away. Otherwise there's a risk that you shut it down too soon leaving the other nodes without a master.

---

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 6:00pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/11 "2019-05-28T18:00:12Z")

</div>

Very helpful. Thank you. Is there any harm in leaving the election node there?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 6:03pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/12 "2019-05-28T18:03:52Z")

</div>

> [@brhardwick](#):
>
> Is there any harm in leaving the election node there?

It depends. Does it have persistent storage or is there a chance it could restart with an empty data directory? If it could lose its data then it'll form a new, empty, cluster when restarted, because of the `cluster.initial_master_nodes` setting, and this could cause you some problems.

---

<div class="post-metadata">

**Author:** ![brhardwick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brhardwick/32/47000_2.png) [@brhardwick](https://discuss.elastic.co/u/brhardwick)\
**Post date:** [May 28, 2019, 6:05pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/13 "2019-05-28T18:05:28Z")

</div>

Ok. That makes sense. Thanks for your help david

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 25, 2019, 6:05pm UTC](https://discuss.elastic.co/t/discovery-with-ecs-dynamic-initial-master-nodes/183149/14 "2019-06-25T18:05:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.