Elasticsearch nodes on aws not showing under single cluster

Elastic Stack Elasticsearch
1

Hi,

We have 3 AWS EC2 servers, where we installed 1 Elasticsearch instance on each of them. We want to have these 3 nodes to be part of the single cluster.

I have exactly followed the steps mentioned in the link https://www.elastic.co/blog/running-elasticsearch-on-aws , but I see only 1 node when I do a “_cluster/health” individually on each of the servers.

curl localhost:9200/_cluster/health?pretty
{
"cluster_name" : "production",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}

Here is the Elasticsearch.yml configuration:

cluster.name: production

node.name: ${HOSTNAME}
node.master: true
node.data: true

bootstrap.mlockall: true

network.host: [machine IPaddress, local]

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [other 2 IP's list inlcuding the current machine IP]

Please let me know what additional configuration changes are required here to have the 3 nodes to be part of single cluster.

Thanks,
Ram

2

As I assume you have 3 master eligible nodes, have you updated minimum_mster_nodes to 2 in order to prevent each of the nodes to form their own cluster?

1 Like
3

Please format your code using </> icon. It will make your post more readable.

Did you check if any firewall is blocking the traffic on port 9300?

1 Like
4

Hi,
I tried to set the discovery.zen.minimum_master_nodes: 2 , but i get the following exception

curl localhost:9200/_cluster/health?pretty
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}

I have set node.master: true and node.data: true on 2 nodes.
I have set node.client: true and node.data: false on the last node.

Am i configuring somehting wrong here?

Thanks,
RK

5

since the servers are AWS EC2 , should i need to install the "cloud-aws" plugin for Elasticsearch and specify the "AccessKey", "SecretKey" and the "Region" in the elasticsearch.yml file ? are these mandatory for the AWS EC2 servers?

6

No but you have to define the ip addresses of other servers in unicast list of nodes.

1 Like
7

yes, i did specify that by doing

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [specifued the IP's of all the 3 nodes including the current machine's IP]

Thanks,
RK

8

Ok. Note that you don't need the first line.

Did you check if any firewall is blocking port 9300?
May be your machines are not in the same VPC and you did not explicitly opened port 9300?

1 Like
9

When i comment out discovery.zen.minimum_master_nodes: 2 , i get the following result when i do a curl http://localhost:9200/_cluster/health?pretty

{
  "cluster_name" : "production",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 25,
  "active_shards" : 25,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 25,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

But when i include the discovery.zen.minimum_master_nodes: 2, i am getting master_not_discovered_exception exception.

I tried the sudo netstat -tulpn | grep 9300 on the shell, it returns a java process running on the port 9300.

I am able to ping other 2 servers from each of the the server.

currently , My overall elasticsearch.yml configuration is

cluster.name: production
node.name: ${HOSTNAME}
node.master: true
#node.data: true
bootstrap.mlockall: true
network.host: [1.2.2.24, _local_]
#network.host: 1.2.2.24
discovery.zen.ping.unicast.hosts: ["1.2.2.24","1.2.1.99", "1.2.2.54"]
discovery.zen.minimum_master_nodes: 2

Note: I have included dummy IP's for security reasons

Thanks,
RK

10

The fact that the java process listens to port 9300 does not indicate that you have connectivity between the nodes. Are you able to telnet from one of the machines to port 9300 on the others?

1 Like
11

The 9300 port was blocked on each of the servers. I've enabled it and its working now.

Thanks a lot for your time !!

Regards,
RK