Adding new node to cluster

Elastic Stack Elasticsearch
1

Hi,
I want to add new node to running cluster. I have two servers on which elastic search 7.8 is setup.

Server # 1
IP address x.x.x.x port 9200
hostname: es1.example.com

Server  # 2
IP Address x.x.x.x port 9200
hostname: es2.example.com

As mentioned in document "Each address can be either an IP address or a hostname which resolves to one or more IP addresses via DNS." so I user hostname in configuration file

Server # 1 elasticsearch.yml
cluster.name: MySureFit-ES
node.name: ${HOSTNAME}
node.master: true
node.data: true
discovery.seed_hosts: ["es1.example.com", "es2.example.com"]

Server # 2 elasticsearch.yml
cluster.name: MySureFit-ES
node.name: ${HOSTNAME}
node.master: false
node.data: true
discovery.seed_hosts: ["es1.example.com", "es2.example.com"]

output of following request _cluster/state/nodes is

{
  "cluster_name" : "MySureFit-ES",
  "cluster_uuid" : "TgTN0IDwRu6TMwAVsgVAyw",
  "nodes" : {
  "e41LhIkzT7uF_LUfO_otSA" : {
  "name" : "v4Dev-Sandbox",
  "ephemeral_id" : "aC7viR8UQcSEnNpJQio9Gw",
  "transport_address" : "127.0.0.1:9300",
  "attributes" : {
    "ml.machine_memory" : "33342156800",
    "xpack.installed" : "true",
    "transform.node" : "true",
    "ml.max_open_jobs" : "20"
   }
  }
 }
}

Can any one help me to resolve this issue.

2

You probably need to check out network.host.

3

Agreed - check your existing nodes for that setting, likely it's 0.0.0.0 which is often simplest (bind to all interfaces) - the default is localhost which means other nodes can't talk to you.

Note if you are on Docker (as I see you are in a Dev Sandbox), you also usually set network.publish_host - see Docker Networking

4

Hi,
I leave network.host to default which is 127.0.0.1:9200. I did not add transport_address.

Should I set server # 1 transport_address to 127.0.0.1:9200 and server # 2 to 127.0.0.1:9300 and both should have same.

I checked with internal IP but no luck. I set network.host to internal IP for both server.

5

I am not on docker I have two server purchase by aws. I do have private IPs with me when I set network.host to private IPs and set discovery.seed_hosts to private IPs still having the same problem.

6

I don't think you can get a cluster working with network.host set to 127.0.0.1 on your other 'working' nodes - set to 0.0.0.0 and they should listen on all IPs. You can also check this via:

netstat -ltnp that will show you what is listening on what - you'll see java listening on port 9200 and 9300, and either on * or 127.0.0.1, etc

Don't think you should set transport_address; I THINK it picks up network.host by default, but you can test.

7

network.host will apply to both REST and binary protocols (9200 and 9300).

8

Hi Guys,
thanks for help after make changes, getting following error

[2020-07-27T14:49:52,424][WARN ][o.e.c.c.ClusterFormationFailureHelper] [v4Dev-Sandbox] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [v4Dev-Sandbox, ES-New] to bootstrap a cluster: have discovered [{v4Dev-Sandbox}{wt5MqhDuQ2i0w7Gqyinugg}{AgzGQPtKQbq5Ni9Q7-F5yQ}{172.31.3.110}{172.31.3.110:9300}{dilmrt}{ml.machine_memory=33342156800, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.31.6.58:9300] from hosts providers and [{v4Dev-Sandbox}{wt5MqhDuQ2i0w7Gqyinugg}{AgzGQPtKQbq5Ni9Q7-F5yQ}{172.31.3.110}{172.31.3.110:9300}{dilmrt}{ml.machine_memory=33342156800, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

Following is elasticsearch.yml

cluster.name: MySureFit-ES
node.name: ${HOSTNAME}
node.master: true
node.data: true
network.host: x.x.x.110
discovery.seed_hosts: ["x.x.x.110", "x.x.x.58"]

Output of the following curl -XGET 'http://x.x.x.110:9200'

{
   "name" : "v4Dev-Sandbox",
   "cluster_name" : "MySureFit-ES",
   "cluster_uuid" : "_na_",
   "version" : {
"number" : "7.8.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65",
"build_date" : "2020-06-14T19:35:50.234439Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Second server

cluster.name: MySureFit-ES
node.name: ${HOSTNAME}
node.master: false
node.data: true
network.host: x.x.x.58
discovery.seed_hosts: ["x.x.x.110", "x.x.x.58"]

Output of following command curl -XGET 'http://x.x.x.58:9200'

{
   "name" : "ES-New",
   "cluster_name" : "MySureFit-ES",
  "cluster_uuid" : "_na_",
  "version" : {
"number" : "7.8.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65",
"build_date" : "2020-06-14T19:35:50.234439Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
 },
 "tagline" : "You Know, for Search"
 }

Error from second server

 [2020-07-27T14:51:22,933][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ES-New] master not discovered yet: have discovered [{ES-New}{xvb57WoeTgml714sQsNvvg}{zYCMuLBtRa2pYNiNW2FwFQ}{172.31.6.58}{172.31.6.58:9300}{dilrt}{ml.machine_memory=8287748096, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {v4Dev-Sandbox}{wt5MqhDuQ2i0w7Gqyinugg}{AgzGQPtKQbq5Ni9Q7-F5yQ}{172.31.3.110}{172.31.3.110:9300}{dilmrt}{ml.machine_memory=33342156800, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}]; discovery will continue using [172.31.3.110:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0

What I did wrong ?

9

What is history here as your 'master' node v4Dev-Sandbox was never part of a cluster, but this whole thread is about having a working cluster and adding a node, so I'm confused.

But your early posting of v4Dev-Sandbox shows a cluster UUID, so it joined then, so how did we lose that? I assume we lost the self-bootstrapping mode with non-local IP, and might need three nodes to build cluster, but this error message is confusing.

@warkolm likely knows much more :wink:

You've kinda changed server names from the original posting of es1, es2 to real names, etc. so I'm a bit lost.

10

According to you that it will not work with 127.0.0.1 so I change it to private IPs. I remove nodes data from /var/lib/elasticsearch/nodes.

When I restart again I am getting this error.

11

But was v4Dev-Sandbox the master before? As it's starting now and not finding a cluster, or at least confused - can we get that node back up and running as the master of a one node cluster again so you have a stable starting point?

12

When I remove

cluster.name: MySureFit-ES
node.name: ${HOSTNAME}
node.master: false
node.data: true
network.host: x.x.x.58
discovery.seed_hosts: ["x.x.x.110", "x.x.x.58"]

node.master

from configuration every thing came back online and now its working and cluster have both nodes.

Thanks for the great help.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.