Installation of ElasticSearch Cluster on AWS EC2 with 3 nodes

[hussain7 (Hussain Kathawala)]

Hello All,

I am a newbie to ElasticSearch Cluster Setup and need help regarding installation. Below is my config for all the 3 nodes. Using ElasticSearch-6.3.2 on Amazon Linux.

Master Config
cluster.name: prod
node.name: master101
node.master: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.Segment: elastic
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true

Slave-1 Config
cluster.name: prod
node.name: slv1
node.master: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.Segment: elastic
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true

Slave-2 Config
cluster.name: prod
node.master: true
node.name: slv2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.Segment: elastic
network.host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
xpack.security.enabled: false
action.destructive_requires_name: true

All of my 3 instances are on a private subnet and Security groups are configured properly and I'm able to ping 3 of my instances vice versa

Getting below error while starting the ElasticSearch service:
Error on Master Node:

[WARN ][o.e.d.z.ZenDiscovery] [master101] not enough master nodes discovered during pinging (found [[Candidate{node={master101}{ssssssssssssss}{ccccccccccccc}{localhost}{127.0.0.1:9300}{ml.machine_memory=16337809408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again   

Error on slv node:

[2018-08-16T08:35:24,619][WARN ][o.e.d.z.ZenDiscovery     ] [slv1] not enough master nodes discovered during pinging (found [[Candidate{node={slv1}{issssssss76DQ}{ccccccccccccwg}{localhost}{127.0.0.1:9300}{ml.machine_memory=16337809408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again 

[spinscale (Alexander Reelsen)]

it seems as if the nodes cannot connect to each other. It also seems you have omitted some configuration options, as it looks that you have the minimum master nodes configured to 2.

can you share more than a single log line.

Also have you followed all the steps in https://www.elastic.co/guide/en/elasticsearch/plugins/6.3/_settings.html and also set up the IAM policy?

[hussain7 (Hussain Kathawala)]

Hi @spinscale
I have already added my access and secret key with proper IAM policy to all of my 3 nodes using the below command:
elasticsearch-keystore add discovery.ec2.access_key
elasticsearch-keystore add discovery.ec2.secret_key
also installed the discovery-ec2 plugin properly.

Also discovery.zen.minimum_master_nodes: 3 is set to 3 on all the three nodes.

Logs from Slave-1 machine:

2018-08-17T11:18:53,687][WARN ][o.e.d.z.ZenDiscovery     ] [slv1] not enough master nodes discovered during pinging (found [[Candidate{node={slv1}{sddsd}{sassaasCr8_f8-FQ}{localhost}{127.0.0.1:9300}{ml.machine_memory=16337809408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [3]), pinging again
[2018-08-17T11:18:56,688][WARN ][o.e.d.z.ZenDiscovery     ] [slv1] not enough master nodes discovered during pinging (found [[Candidate{node={slv1}{asassbfUK76DQ}{sasasasRfG5KCr8_f8-FQ}{localhost}{127.0.0.1:9300}{ml.machine_memory=16337809408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [3]), pinging again

Logs from Slave-2 machine:

[2018-08-17T11:20:38,493][WARN ][o.e.d.z.ZenDiscovery     ] [slv2] not enough master nodes discovered during pinging (found [[Candidate{node={slv2}{ssssssss_Qn-0yYubPx-Hng}{sssss1wSu6PgJt0MDmCMQ}{localhost}{127.0.0.1:9300}{ml.machine_memory=16337809408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [3]), pinging again
[2018-08-17T11:20:41,494][WARN ][o.e.d.z.ZenDiscovery     ] [slv2] not enough master nodes discovered during pinging (found [[Candidate{node={slv2}{sssssss0yYubPx-Hng}{ssssssu6PgJt0MDmCMQ}{localhost}{127.0.0.1:9300}{ml.machine_memory=16337809408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [3]), pinging again

Please let me know if you need any other info from me.