Add a coordinating/client node to my existing Elasticsearch cluster

I already have a 3 node Elastic search cluster setup with Kibana installed on all the nodes..

PROBLEM: The problem is when we stop master node in Elastic search, the entire Kibana goes down and is inaccessible.

What I want is to install an instance of elasticsearch on any of the node in elasticsearch cluster and make it as coordinating/client node and make kibana point to this coordinating node.

Configurations in the nodes is given below:

NODE1:
cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: true

node.data: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

http.port: 9200

network.host: node1

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

NODE2:

cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: true

node.data: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

http.port: 9200

network.host: node2

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

NODE 3:

cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: true

node.data: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

http.port: 9200

network.host: node3

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

Added a new node (coordinating or client node) on NODE3 only. cluster name remains same. But configuring this elasticsearch to run on different port

cluster.name: es-prod

node.name: ${HOSTNAME}

node.master: false

node.data: false

node.ingest: false

path.data: /home/user/elasticsearch6.5.4

path.logs: /home/user/elasticsearch6.5.4

bootstrap.memory_lock: true

http.port: 9202

network.host:

transport.host:

transport.tcp.port: 9302

discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]

discovery.zen.minimum_master_nodes: 2

Now in all the kibana.yml in all 3 nodes, changed elasticsearch.url in elasticsearch.yml file
to i.e coordinating/client node

When I tried to start this coordinating node on a different port, It is showing errors as:

[2019-02-01T00:00:57,580][INFO ][o.e.b.BootstrapChecks ] [elasticsearch] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-02-01T00:01:00,606][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:03,608][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:06,609][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:09,610][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:12,611][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:15,612][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:18,613][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:21,615][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:24,616][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:27,607][WARN ][o.e.n.Node ] [elasticsearch] timed out while waiting for initial discovery state - timeout: 30s
[2019-02-01T00:01:27,617][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:27,619][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [elasticsearch] publish_address {10.21.0.127:9202}, bound_addresses {10.21.0.127:9202}
[2019-02-01T00:01:27,620][INFO ][o.e.n.Node ] [elasticsearch] started
[2019-02-01T00:01:30,618][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:33,619][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:36,621][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:39,622][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:42,623][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:45,624][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again
[2019-02-01T00:01:48,625][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch] not enough master nodes discovered during pinging (found [], but needed [2]), pinging again

Not able to start this node and add it to my existing cluster..

The transport.tcp.port setting applies to every node in discovery.zen.ping.unicast.hosts too, unless they have an explicit port. You need to say discovery.zen.ping.unicast.hosts: ["node1:9300", "node2:9300", "node3:9300"] to override this behaviour.

Hey @DavidTurner

I tried the approach as you described.. In all the elasticsearch.yml (including coordinating node as well) file I changed
discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3:"]

to

discovery.zen.ping.unicast.hosts: ["node1:9300", "node2:9300", "node3:9300"]

But Still I am getting errors like:
[2019-02-01T01:33:28,914][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch] failed to send join request to master [{FJ7ng38UTtunkayrISGy8g}{hANyvC0tQ7CAyFD78gOhNA}{ml.machine_memory=16647131136, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[internal:discovery/zen/join]]; nested: ConnectTransportException[[elasticsearch]connect_exception]; nested: IOException[No route to host]; ]

This is a different error, which is progress :slight_smile: It looks like a networking issue.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.