Install Cluster of Elasticsearch on EC2 AWS

Elastic Stack Elasticsearch
1

Hi, currently I'm trying to deploy a cluster with 3 nodes on EC2 AWS. The configuration is the following:

#master node
cluster.name: woombat
node.name: node1
node.master: true
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2


#node 2
cluster.name: woombat
node.name: node2
node.master: true
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2

In node1 i'm getting this:

[2020-02-22T15:05:42,230][INFO ][o.e.c.c.Coordinator ] [node1] cluster UUID [L0Fkb2CcTGCGRc2ZP1jZew]
[2020-02-22T15:05:42,363][INFO ][o.e.c.s.MasterService ] [node1] elected-as-master ([1] nodes joined)[{node1}{IFL6k3JmTy-6MMbEpMF6YA}{uQUqHv63RXKPfPYXL3pG2w}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 7, version: 30, reason: master node changed {previous , current [{node1}{IFL6k3JmTy-6MMbEpMF6YA}{uQUqHv63RXKPfPYXL3pG2w}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-02-22T15:05:42,510][INFO ][o.e.c.s.ClusterApplierService] [node1] master node changed {previous , current [{node1}{IFL6k3JmTy-6MMbEpMF6YA}{uQUqHv63RXKPfPYXL3pG2w}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20}]}, term: 7, version: 30, reason: Publication{term=7, version=30}

In node2 i'm getting this:

[2020-02-22T15:00:15,696][INFO ][o.e.c.c.Coordinator ] [node2] cluster UUID [rI_bmBNzTW2ynaNmi5dPjg]
[2020-02-22T15:00:15,841][INFO ][o.e.c.s.MasterService ] [node2] elected-as-master ([1] nodes joined)[{node2}{zn-PL4x8Qrmvqj9veBgM5Q}{pk5SXza_TL-u37gkaZBWxA}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 10, version: 40, reason: master node changed {previous , current [{node2}{zn-PL4x8Qrmvqj9veBgM5Q}{pk5SXza_TL-u37gkaZBWxA}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-02-22T15:00:15,999][INFO ][o.e.c.s.ClusterApplierService] [node2] master node changed {previous , current [{node2}{zn-PL4x8Qrmvqj9veBgM5Q}{pk5SXza_TL-u37gkaZBWxA}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20}]}, term: 10, version: 40, reason: Publication{term=10, version=40}

In both nodes i'm getting this output with the warning:

[node1] Exception while retrieving instance list from AWS API: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@57915fb5: access denied ("java.io.FilePermission" "/home/ec2-user/.aws/credentials" "read"), com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@45bfaad5: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]

Also, The cluster Id of both is different, I think is because they are not finding themselves. I installed the discovery-ec2 plugin. How can i fix this?

2

Yes, that means you have two (or more) separate clusters which means you did not configure cluster bootstrapping correctly. You cannot join separate clusters together, so you must start again.

Elasticsearch will use the access key given in discovery.ec2.access_key if set, or the instance role if not. It looks like you have not specified an access key, but nor have you configured your instances to have an instance role.

1 Like
3

Thanks, The warning about AWS credentials after attach the policy:

{
  "Statement": [
    {
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

On the other hand, I tried with:

discovery.seed_hosts: [172.31.50.181,172.31.56.8]
cluster.initial_master_nodes: ["node1","node2"]

But it didn't work :frowning_face: each node it's creating it's own cluster.

Any idea?

Thanks!

4

Did you wipe the data directories as per the instructions I linked above? If so, can you share the logs again?

1 Like
5

After wipe the data directories (deletting the folder inside data directorie) i'm getting the following warning on node1:

[2020-02-24T12:40:47,272][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1, node2] to bootstrap a cluster: have discovered ; discovery will continue using [172.31.50.181:9300, 172.31.56.8:9300] from hosts providers and [{node1}{y4SzqdUNSCC0IKtfMnmIxA}{uDspm9k7QMaYkNZnrPy5QA}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

and in node2:

[2020-02-24T12:40:33,547][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1, node2] to bootstrap a cluster: have discovered ; discovery will continue using [172.31.50.181:9300, 172.31.56.8:9300] from hosts providers and [{node2}{hTyeJGh9R3KM07pFxue4hQ}{EarDSCwQTr-ACNGhH63-nA}{localhost}{127.0.0.1:9300}{ml.machine_memory=2088099840, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

The current configuration is:

cluster.name: woombat
node.name: node1
node.master: true
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2

discovery.seed_hosts: [172.31.50.181,172.31.56.8]
cluster.initial_master_nodes: ["node1","node2"]

Any idea how to fix?
Thanks

6

Both nodes have address 127.0.0.1:9300 because you have set transport.host: localhost. This address is inaccessible to remote nodes.

1 Like
7

After delete transport.host: localhost from the config file the nodes never strat, here are the logs:

[2020-02-24T13:12:34,829][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node1] [controller/3838] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV
[2020-02-24T13:12:35,910][INFO ][o.e.d.DiscoveryModule ] [node1] using discovery type [zen] and seed hosts providers [settings, ec2]
[2020-02-24T13:12:37,380][INFO ][o.e.n.Node ] [node1] initialized
[2020-02-24T13:12:37,386][INFO ][o.e.n.Node ] [node1] starting ...
[2020-02-24T13:12:37,620][INFO ][o.e.t.TransportService ] [node1] publish_address {172.31.56.8:9300}, bound_addresses {[::]:9300}
[2020-02-24T13:12:37,648][INFO ][o.e.b.BootstrapChecks ] [node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [3] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: memory locking requested for elasticsearch process but memory is not locked
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2020-02-24T13:12:37,670][INFO ][o.e.n.Node ] [node1] stopping ...
[2020-02-24T13:12:37,687][INFO ][o.e.n.Node ] [node1] stopped
[2020-02-24T13:12:37,688][INFO ][o.e.n.Node ] [node1] closing ...
[2020-02-24T13:12:37,733][INFO ][o.e.n.Node ] [node1] closed
[2020-02-24T13:12:37,773][INFO ][o.e.x.m.p.NativeController] [node1] Native controller process has stopped - no new native processes can be started

Any idea?
Thanks

8

Yes, the things you need to address are listed in the log:

See Bootstrap Checks | Elasticsearch Guide [8.11] | Elastic for more detail.

1 Like
9

Thanks. After make this changes on both nodes:

sudo nano /etc/security/limits.conf
#adding this:
ec2-user  -  nofile  65535

sudo nano /etc/sysctl.conf
#adding this:
vm.max_map_count = 262144
sudo reboot

#sudo nano config/elasticsearch.yml
cluster.name: my_cluster
node.name: node1
node.master: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ec2discovery: elk
network.host: 0.0.0.0
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true
discovery.zen.minimum_master_nodes: 2
discovery.seed_hosts: [172.31.50.181,172.31.56.8]
cluster.initial_master_nodes: ["node1","node2"]

Solved!

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.