How to create elastic search cluster on AWS-EC2 using ES-6.5.2

Elastic Stack Elasticsearch
1

Hi am creating cluster with 3 nodes making 2 nodes as master and data, 1 is data node

  1. discovery.zen.ping.unicast.hosts: ["master_node1:9300", "master_node2:9300", "datanode:9300"]
    i have added like this in three nodes
  2. node.name: node-1
    node.master: true
    node.data: true
    node.ingest: true
    discovery.zen.hosts_provider: ec2

network.host: 0.0.0.0

these are the configurations

but i am able to see only one node i.e. master

result:

{
"cluster_name" : "Test_ES",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

could you please help.

2

That's a bad idea. You should have 3 master nodes.

If you define manually the unicast settings then remove this:

discovery.zen.hosts_provider: ec2
3

If you have 2 or 3 master eligible nodes in the cluster, you must also set discovery.zen.minimum_master_nodes to 2 as per these guidelines.

4

Hey David Pilato and Christian_Dahlqvist

Thanks for reply

1.I just installed the below plugin and i wrote this line "discovery.zen.hosts_provider: ec2"

cd /usr/share/elasticsearch/bin

sudo ./elasticsearch-plugin install discovery-ec2

  1. if i add "discovery.zen.minimum_master_nodes to 2"

the cluster will say there is no master node

5
  1. if i add " discovery.zen.minimum_master_nodes to 2 "

the cluster will say there is no master node

Because the connection between nodes has not established.

I opened ports 9300, 9200 too

6

node-1

node.name: node-1
node.master: true
node.data: true
node.ingest: true
#discovery.zen.hosts_provider: ec2
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["node-1", "node-2", "node-3"]
#discovery.zen.minimum_master_nodes: 2

====================================================================
node-2

node.name: node-2
node.master: true
node.data: true
node.ingest: true
#discovery.zen.hosts_provider: ec2
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["node-1", "node-2", "node-3"]
#discovery.zen.minimum_master_nodes: 2

==================================================================
node-3

node.name: node-3
node.master: false
node.data: true
node.ingest: true
#discovery.zen.hosts_provider: ec2
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["node-1", "node-2", "node-3"]
#discovery.zen.minimum_master_nodes: 2

if i curl from node-1

OUTPUT:

curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "Test_ES",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

Please help me to add nodes to this cluster.
Thanks.

7

As David pointed out, having 2 master-eligible nodes is bad and you should make all 3 master-eligible.

You should not have this commented out.

Does this resolve to the IP addresses of the nodes in the cluster? Are you able to telnet to port 9300 between the hosts?

8

I made all three as master nodes
and "discovery.zen.minimum_master_nodes: 2" added that line

telnet IP 9300
Trying IP...
Connected to IP.
Escape character is '^]'.

now OUTPUT:
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}

9

this is the node-1 log

[2018-12-13T00:39:17,007][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{KaO-RmXBQYKDukApaHOW8g}{L9ODM6TdTUmp5JNlyM7mKg}{IP}{IP:9300}{ml.machine_memory=4135096320, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node-2][IP:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {node-1}{KaO-RmXBQYKDukApaHOW8g}{RB-Oa2GNTyON1wy5UC0aPQ}{IP}{IP:9300}{ml.machine_memory=4135108608, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, found existing node {node-2}{KaO-RmXBQYKDukApaHOW8g}{L9ODM6TdTUmp5JNlyM7mKg}{IP}{IP:9300}{ml.machine_memory=4135096320, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} with the same id but is a different node instance]; ]

10

Are you specifying IP or hostname in discovery.zen.ping.unicast.hosts?

11

now i changed to IP

12

Same output
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}

No luck as of now

LOGS:

[2018-12-13T01:16:29,714][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{KaO-RmXBQYKDukApaHOW8g}{L9ODM6TdTUmp5JNlyM7mKg}{IP}{IP:9300}{ml.machine_memory=4135096320, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node-2][IP:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {node-1}{KaO-RmXBQYKDukApaHOW8g}{RB-Oa2GNTyON1wy5UC0aPQ}{IP}{IP:9300}{ml.machine_memory=4135108608, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, found existing node {node-2}{KaO-RmXBQYKDukApaHOW8g}{L9ODM6TdTUmp5JNlyM7mKg}{IP}{IP:9300}{ml.machine_memory=4135096320, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} with the same id but is a different node instance]; ]

Any help?

13

How did you install Elasticsearch on the hosts? Did you by any chance just copy the disks or directories instead of installing from scratch?

14

I installed from scratch on node1, after I took image and launched two more instances, After I cahnged config files on each instance.

15

  1. So you mean to say ID's are same, do I need to install from scratch on each server?

  2. Also de we need any plugins for Aws specifically?

16

I believe the ID is generated when the node first starts up, so I assume you started Elasticsearch on the node before you took the image? This would in that case explain the issue. restarting while clearing the data directory and should force a new ID to be generated. There may be easier ways to correct this, but I have not encountered this before.

17

Also de we need any plugins for Aws specifically?

If you want to use automatic discovery and configure network.host based on _ec2_ IP addresses, yes. Otherwise just remove the discovery-ec2 plugin.

18

Thanks Christian and David
Error was same Id, on all nodes, because of installation.

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.