Add a coordinating/client node to my existing Elasticsearch cluster

Elastic Stack Elasticsearch
1

I already have a 3 node Elastic search cluster setup with Kibana installed on all the nodes..

PROBLEM: The problem is when we stop master node in Elastic search, the entire Kibana goes down and is inaccessible.

What I want is to install an instance of elasticsearch on any of the node in elasticsearch cluster and make it as coordinating/client node and make kibana point to this coordinating node.

Configurations in the nodes is given below:

NODE1:
cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: true

node.data: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

http.port: 9200

network.host: node1

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

NODE2:

cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: true

node.data: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

http.port: 9200

network.host: node2

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

NODE 3:

cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: true

node.data: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

http.port: 9200

network.host: node3

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

Added a new node (coordinating or client node) on NODE3 only. cluster name remains same. But configuring this elasticsearch to run on different port

cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: false

node.data: false

node.ingest: false

path.data: /home/user/elasticsearch6.5.4

path.logs: /home/user/elasticsearch6.5.4

bootstrap.memory_lock: true

http.port: 9202

network.host:

transport.host:

transport.tcp.port: 9302

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

Now in all the kibana.yml in all 3 nodes, changed elasticsearch.url in elasticsearch.yml file
to i.e coordinating/client node

When I tried to start this coordinating node on a different port, It is showing errors as:

[2019-02-01T00:00:57,580][INFO ][o.e.b.BootstrapChecks ] [elasticsearch] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-02-01T00:01:00,606][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:03,608][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:06,609][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:09,610][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:12,611][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:15,612][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:18,613][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:21,615][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:24,616][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:27,607][WARN ][o.e.n.Node ] [elasticsearch] timed out while waiting for initial discovery state - timeout: 30s
[2019-02-01T00:01:27,617][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:27,619][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [elasticsearch] publish_address {10.21.0.127:9202}, bound_addresses {10.21.0.127:9202}
[2019-02-01T00:01:27,620][INFO ][o.e.n.Node ] [elasticsearch] started
[2019-02-01T00:01:30,618][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:33,619][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:36,621][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:39,622][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:42,623][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:45,624][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:48,625][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again

Not able to start this node and add it to my existing cluster..

2

The transport.tcp.port setting applies to every node in discovery.zen.ping.unicast.hosts too, unless they have an explicit port. You need to say discovery.zen.ping.unicast.hosts: ["node1:9300", "node2:9300", "node3:9300"] to override this behaviour.

3

Hey @DavidTurner

I tried the approach as you described.. In all the elasticsearch.yml (including coordinating node as well) file I changed
discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3:"]

to

discovery.zen.ping.unicast.hosts: ["node1:9300", "node2:9300", "node3:9300"]

But Still I am getting errors like:
[2019-02-01T01:33:28,914][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch] failed to send join request to master [{FJ7ng38UTtunkayrISGy8g}{hANyvC0tQ7CAyFD78gOhNA}{ml.machine_memory=16647131136, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[internal:discovery/zen/join]]; nested: ConnectTransportException[[elasticsearch]connect_exception]; nested: IOException[No route to host]; ]

4

This is a different error, which is progress :slight_smile: It looks like a networking issue.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.