# Install Cluster of Elasticsearch on EC2 AWS

**URL:** <https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494>\
**Category:** Elasticsearch\
**Created:** [February 22, 2020, 3:08pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494 "2020-02-22T15:08:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Miguel\_Plazas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel_plazas/32/40038_2.png) [@Miguel\_Plazas](https://discuss.elastic.co/u/Miguel_Plazas)\
**Post date:** [February 22, 2020, 3:08pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/1 "2020-02-22T15:08:16Z")

</div>

Hi, currently I'm trying to deploy a cluster with 3 nodes on EC2 AWS. The configuration is the following:

```
#master node
cluster.name: woombat
node.name: node1
node.master: true
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2

#node 2
cluster.name: woombat
node.name: node2
node.master: true
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2

```

In node1 i'm getting this:

> [2020-02-22T15:05:42,230][INFO][o.e.c.c.Coordinator] [node1] cluster UUID [L0Fkb2CcTGCGRc2ZP1jZew]  
> [2020-02-22T15:05:42,363][INFO][o.e.c.s.MasterService] [node1] elected-as-master ([1] nodes joined)[{node1}{IFL6k3JmTy-6MMbEpMF6YA}{uQUqHv63RXKPfPYXL3pG2w}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20} elect leader, _BECOME\_MASTER\_TASK_, _FINISH\_ELECTION_], term: 7, version: 30, reason: master node changed {previous , current [{node1}{IFL6k3JmTy-6MMbEpMF6YA}{uQUqHv63RXKPfPYXL3pG2w}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20}]}  
> [2020-02-22T15:05:42,510][INFO][o.e.c.s.ClusterApplierService] [node1] master node changed {previous , current [{node1}{IFL6k3JmTy-6MMbEpMF6YA}{uQUqHv63RXKPfPYXL3pG2w}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20}]}, term: 7, version: 30, reason: Publication{term=7, version=30}

In node2 i'm getting this:

> [2020-02-22T15:00:15,696][INFO][o.e.c.c.Coordinator] [node2] cluster UUID [rI\_bmBNzTW2ynaNmi5dPjg]  
> [2020-02-22T15:00:15,841][INFO][o.e.c.s.MasterService] [node2] elected-as-master ([1] nodes joined)[{node2}{zn-PL4x8Qrmvqj9veBgM5Q}{pk5SXza\_TL-u37gkaZBWxA}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20} elect leader, _BECOME\_MASTER\_TASK_, _FINISH\_ELECTION_], term: 10, version: 40, reason: master node changed {previous , current [{node2}{zn-PL4x8Qrmvqj9veBgM5Q}{pk5SXza\_TL-u37gkaZBWxA}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20}]}  
> [2020-02-22T15:00:15,999][INFO][o.e.c.s.ClusterApplierService] [node2] master node changed {previous , current [{node2}{zn-PL4x8Qrmvqj9veBgM5Q}{pk5SXza\_TL-u37gkaZBWxA}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20}]}, term: 10, version: 40, reason: Publication{term=10, version=40}

In both nodes i'm getting this output with the warning:

> [node1] Exception while retrieving instance list from AWS API: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS\_ACCESS\_KEY\_ID (or AWS\_ACCESS\_KEY) and AWS\_SECRET\_KEY (or AWS\_SECRET\_ACCESS\_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@57915fb5: access denied ("java.io.FilePermission" "/home/ec2-user/.aws/credentials" "read"), com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@45bfaad5: The requested metadata is not found at [http://169.254.169.254/latest/meta-data/iam/security-credentials/](http://169.254.169.254/latest/meta-data/iam/security-credentials/)]

Also, The cluster Id of both is different, I think is because they are not finding themselves. I installed the discovery-ec2 plugin. How can i fix this?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 22, 2020, 3:49pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/2 "2020-02-22T15:49:29Z")

</div>

> [@Miguel\_Plazas](#):
>
> The cluster Id of both is different

Yes, that means you have two (or more) separate clusters which means you did not [configure cluster bootstrapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#initial_master_nodes) correctly. [You cannot join separate clusters together, so you must start again](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-joining).

> [@Miguel\_Plazas](#):
>
> Unable to load AWS credentials from any provider in the chain

Elasticsearch will use the access key given in `discovery.ec2.access_key` if set, or the instance role if not. It looks like you have not specified an access key, but nor have you configured your instances to have an instance role.

---

<div class="post-metadata">

**Author:** ![Miguel\_Plazas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel_plazas/32/40038_2.png) [@Miguel\_Plazas](https://discuss.elastic.co/u/Miguel_Plazas)\
**Post date:** [February 22, 2020, 10:02pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/3 "2020-02-22T22:02:17Z")

</div>

Thanks, The warning about AWS credentials after attach the policy:

```
{
  "Statement": [
    {
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

```

On the other hand, I tried with:

```
discovery.seed_hosts: [172.31.50.181,172.31.56.8]
cluster.initial_master_nodes: ["node1","node2"]

```

But it didn't work ☹ each node it's creating it's own cluster.

Any idea?

Thanks!

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 23, 2020, 8:25am UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/4 "2020-02-23T08:25:40Z")

</div>

> [@Miguel\_Plazas](#):
>
> each node it's creating it's own cluster.

Did you wipe the data directories as per the instructions I linked above? If so, can you share the logs again?

---

<div class="post-metadata">

**Author:** ![Miguel\_Plazas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel_plazas/32/40038_2.png) [@Miguel\_Plazas](https://discuss.elastic.co/u/Miguel_Plazas)\
**Post date:** [February 24, 2020, 12:56pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/5 "2020-02-24T12:56:30Z")

</div>

After wipe the data directories (deletting the folder inside data directorie) i'm getting the following warning on node1:

> [2020-02-24T12:40:47,272][WARN][o.e.c.c.ClusterFormationFailureHelper] [node1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1, node2] to bootstrap a cluster: have discovered ; discovery will continue using [172.31.50.181:9300, 172.31.56.8:9300] from hosts providers and [{node1}{y4SzqdUNSCC0IKtfMnmIxA}{uDspm9k7QMaYkNZnrPy5QA}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

and in node2:

> [2020-02-24T12:40:33,547][WARN][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1, node2] to bootstrap a cluster: have discovered ; discovery will continue using [172.31.50.181:9300, 172.31.56.8:9300] from hosts providers and [{node2}{hTyeJGh9R3KM07pFxue4hQ}{EarDSCwQTr-ACNGhH63-nA}{localhost}{127.0.0.1:9300}{ml.machine\_memory=2088099840, xpack.installed=true, ml.max\_open\_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

The current configuration is:

```
cluster.name: woombat
node.name: node1
node.master: true
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2

discovery.seed_hosts: [172.31.50.181,172.31.56.8]
cluster.initial_master_nodes: ["node1","node2"]

```

Any idea how to fix?  
Thanks

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 24, 2020, 1:04pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/6 "2020-02-24T13:04:29Z")

</div>

Both nodes have address `127.0.0.1:9300` because you have set `transport.host: localhost`. This address is inaccessible to remote nodes.

---

<div class="post-metadata">

**Author:** ![Miguel\_Plazas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel_plazas/32/40038_2.png) [@Miguel\_Plazas](https://discuss.elastic.co/u/Miguel_Plazas)\
**Post date:** [February 24, 2020, 1:15pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/7 "2020-02-24T13:15:21Z")

</div>

After delete `transport.host: localhost` from the config file the nodes never strat, here are the logs:

> [2020-02-24T13:12:34,829][INFO][o.e.x.m.p.l.CppLogMessageHandler] [node1] [controller/3838] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV  
> [2020-02-24T13:12:35,910][INFO][o.e.d.DiscoveryModule] [node1] using discovery type [zen] and seed hosts providers [settings, ec2]  
> [2020-02-24T13:12:37,380][INFO][o.e.n.Node] [node1] initialized  
> [2020-02-24T13:12:37,386][INFO][o.e.n.Node] [node1] starting ...  
> [2020-02-24T13:12:37,620][INFO][o.e.t.TransportService] [node1] publish\_address {172.31.56.8:9300}, bound\_addresses {[::]:9300}  
> [2020-02-24T13:12:37,648][INFO][o.e.b.BootstrapChecks] [node1] bound or publishing to a non-loopback address, enforcing bootstrap checks  
> ERROR: [3] bootstrap checks failed  
> [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]  
> [2]: memory locking requested for elasticsearch process but memory is not locked  
> [3]: max virtual memory areas vm.max\_map\_count [65530] is too low, increase to at least [262144]  
> [2020-02-24T13:12:37,670][INFO][o.e.n.Node] [node1] stopping ...  
> [2020-02-24T13:12:37,687][INFO][o.e.n.Node] [node1] stopped  
> [2020-02-24T13:12:37,688][INFO][o.e.n.Node] [node1] closing ...  
> [2020-02-24T13:12:37,733][INFO][o.e.n.Node] [node1] closed  
> [2020-02-24T13:12:37,773][INFO][o.e.x.m.p.NativeController] [node1] Native controller process has stopped - no new native processes can be started

Any idea?  
Thanks

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 24, 2020, 1:17pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/8 "2020-02-24T13:17:41Z")

</div>

Yes, the things you need to address are listed in the log:

> [@Miguel\_Plazas](#):
>
> ```plaintext
> ERROR: [3] bootstrap checks failed
> [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
> [2]: memory locking requested for elasticsearch process but memory is not locked
> [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
> 
> ```

See [Bootstrap Checks | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html) for more detail.

---

<div class="post-metadata">

**Author:** ![Miguel\_Plazas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel_plazas/32/40038_2.png) [@Miguel\_Plazas](https://discuss.elastic.co/u/Miguel_Plazas)\
**Post date:** [February 24, 2020, 2:45pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/9 "2020-02-24T14:45:02Z")

</div>

Thanks. After make this changes on both nodes:

```
sudo nano /etc/security/limits.conf
#adding this:
ec2-user - nofile 65535

sudo nano /etc/sysctl.conf
#adding this:
vm.max_map_count = 262144
sudo reboot

#sudo nano config/elasticsearch.yml
cluster.name: my_cluster
node.name: node1
node.master: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2
discovery.seed_hosts: [172.31.50.181,172.31.56.8]
cluster.initial_master_nodes: ["node1","node2"]

```

Solved!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2020, 2:45pm UTC](https://discuss.elastic.co/t/install-cluster-of-elasticsearch-on-ec2-aws/220494/10 "2020-03-23T14:45:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.