Node management and values

Elastic Stack Elasticsearch
1

Howdy,
I'm new to elasticsearch, and I've tried to look up FAQs and things but no luck so far, so.... here goes.

We have a 3+10 cluster I get to inherit administration of.
3 master nodes, 10 data.
I need to add some new, higher disk capacity nodes into the cluster. and then eventually cycle through all the old nodes, upgrading disks for each of them.
its a production cluster, so.... yeah :-/

I notice that Every single node currently has an elasticsearch.yml file. They are practically identical across all nodes. Each define

gateway: 
  expected_nodes: 10

So.. when I want to add nodes, do I need to go change the file on every single node, and make sure it has the correct value?

secondly.. every single yml config file has

discovery: 
  zen: 
    minimum_master_nodes: 2
    ping: 
      unicast: 
        hosts: 
             - host01
             - host02
      ...

Since this is unicast, not multicast.. does this mean that, again, I have to go to every single host and change the file to identify new hosts that I want to add to the cluster?

I'm confused, because all the web search I do about "how to add a node to elasticsearch cluster" just say to create the config on the new node, start the new node, and it just AutoMagically happens. There's no mention of having to go edit files on any of the existing nodes.

but if so, what are the values in the elasticsearch.yml file for???

2

You don't have to, no. The gateway.expected_nodes setting is described here and can safely be set to a number lower than the number of nodes in the cluster. It is optional, only has an effect on the master-eligible nodes, and only when completely restarting the cluster. I think I would leave it at 10 for now and then adjust it later if needed.

Really you should only be listing the master-eligible nodes under discovery.zen.ping.unicast.hosts. That way you don't need to adjust it when you add a master-ineligible node like a data node.

3

Hm...
so.. in theory, I should be able to add the new node, listing just the masters.
but .. it isnt working.
and its giving me a bogus "no route to host" error, when tcpdump and telnet both says that traffic IS actually flowing from the new node to the current master node.

Log file entries:

NFO ][node ] [oc2esdata11-oc2] starting ...
[2019-09-09 16:11:38,835][INFO ][transport ] [oc2esdata11-oc2] publish_address {1.2.3.4:9300}, bound_addresses {1.2.3.4:9300}
[2019-09-09 16:11:38,843][INFO ][discovery ] [oc2esdata11-oc2] elasticsearch_logstash/eKYDs3CIRQOaniiJyLVGUg
[2019-09-09 16:11:41,901][INFO ][discovery.zen ] [oc2esdata11-oc2] failed to send join request to master [{oc2esnode02-oc2}{SLDhR7foQwuCpwp5NsecUg}{6.7.8.9}{6.7.8.9:9300}{rack=vm, data=false, master=true}], reason [RemoteTransportException[[oc2esnode02-oc2][6.7.8.9:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[oc2esdata11-oc2][1.2.3.4:9300] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[no_route_to_host_exception: No route to host]; ]

telnet test:

[root@oc2esdata11 oc2]# telnet 6.7.8.9 9300
Trying 6.7.8.9...
Connected to 6.7.8.9.
Escape character is '^]'.

4

PS: googled a "fix" for this, but new node does have this defined already:
network:
host: enp4s0f0:ipv4

and that IS the actual active network interface name.
enp4s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

5

oh. I thought you only needed port 9300 to the master. but apparently you also need a connection back on 9300,even if NOT a master.
So I guess I'll have to get the firewalling tweaked for that!

1 Like
6

Yes that's right, this exception is the master trying (and failing) to connect back to the new node, which means that the new node got through to the master ok.

7

clearer error messages would have been nice :-/

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.