Difficulty adding nodes

Brett_Blackham · June 12, 2019, 12:11am

I've been following several tutorials/examples but I am not able to get elastic to see other nodes.

I have 3 VMs running Ubuntu 18.04 LTS

/etc/hosts
10.192.10.61 elastic1
10.192.10.62 elastic2
10.192.10.63 elastic3

I can telnet to port 9200 and 9300 on all three servers. But I can never get it to show more than one node. I imagine I'm missing something silly in my config.

root@elastic1:/var/log# curl -XGET 'http://10.192.10.61:9200/_cluster/health?pretty'
{
  "cluster_name" : "the-cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : "NaN"
}

Here are the config files

Tail of the cluster log
[2019-06-12T00:03:06,983][INFO ][o.e.p.PluginsService ] [elastic1] loaded module [x-pack-security]
[2019-06-12T00:03:06,984][INFO ][o.e.p.PluginsService ] [elastic1] loaded module [x-pack-sql]
[2019-06-12T00:03:06,984][INFO ][o.e.p.PluginsService ] [elastic1] loaded module [x-pack-watcher]
[2019-06-12T00:03:06,985][INFO ][o.e.p.PluginsService ] [elastic1] no plugins loaded
[2019-06-12T00:03:11,566][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic1] [controller/5736] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV
[2019-06-12T00:03:12,369][INFO ][o.e.d.DiscoveryModule ] [elastic1] using discovery type [zen] and seed hosts providers [settings]
[2019-06-12T00:03:13,324][INFO ][o.e.n.Node ] [elastic1] initialized
[2019-06-12T00:03:13,325][INFO ][o.e.n.Node ] [elastic1] starting ...
[2019-06-12T00:03:13,490][INFO ][o.e.t.TransportService ] [elastic1] publish_address {10.192.10.61:9300}, bound_addresses {10.192.10.61:9300}
[2019-06-12T00:03:13,499][INFO ][o.e.b.BootstrapChecks ] [elastic1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-12T00:03:13,507][INFO ][o.e.c.c.Coordinator ] [elastic1] cluster UUID [8V5kNevdQDuGgqMCCjRg5A]
[2019-06-12T00:03:13,667][INFO ][o.e.c.s.MasterService ] [elastic1] elected-as-master ([1] nodes joined)[{elastic1}{DpJg-1UISCSBF_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine_memory=16819339264, xpack.installed=true, ml.max_ope
n_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 37, version: 95, reason: master node changed {previous , current [{elastic1}{DpJg-1UISCSBF_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine_memory=16819339264,
xpack.installed=true, ml.max_open_jobs=20}]}
[2019-06-12T00:03:13,729][INFO ][o.e.c.s.ClusterApplierService] [elastic1] master node changed {previous , current [{elastic1}{DpJg-1UISCSBF_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine_memory=16819339264, xpack.installed=true,
ml.max_open_jobs=20}]}, term: 37, version: 95, reason: Publication{term=37, version=95}
[2019-06-12T00:03:13,806][INFO ][o.e.h.AbstractHttpServerTransport] [elastic1] publish_address {10.192.10.61:9200}, bound_addresses {10.192.10.61:9200}
[2019-06-12T00:03:13,806][INFO ][o.e.n.Node ] [elastic1] started
[2019-06-12T00:03:45,840][INFO ][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][young][32][3] duration [778ms], collections [1]/[1.4s], total [778ms]/[844ms], memory [582.4mb]->[212.8mb]/[1.9gb], all_pools {[young] [527.1mb]->[19.5mb]/[532.5mb]}{[survivor] [55.3mb]
->[56.1mb]/[66.5mb]}{[old] [0b]->[138.9mb]/[1.3gb]}
[2019-06-12T00:03:45,841][WARN ][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][32] overhead, spent [778ms] collecting in the last [1.4s]

Can anyone see what I'm missing?

shin-higuchi · June 12, 2019, 3:26am

Hi,

I guess you need to configure "cluster.initial_master_nodes" in your elasticsearch.yml.

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html

Thanks.

Brett_Blackham · June 12, 2019, 1:23pm

Thanks for looking. I have tried it with cluster.initial_master_nodes commented out. Also with just elastict1, and with elastic2 and elastic3. Same results every time.

The only setting that seems to change the _cluster/health is gateway.recover_after_nodes. Setting it to 2 will start but with a "status": "red". A value of 1 (or commented out) will start with a "status": "green".

I found someone else that was getting this same issue. In fact, I recognize his configs. He used one of the 5 different example/tutorials I saw. So, I'm not the only one running into this.
He had the same, or similar, issue

Hari.v said he got his cluster to work by adding "node.master: true , node.data: true for all the 3 nodes". I modified my config to mach his (again), but now I have three ES servers each running as individual (single node) clusters.

Below is the log file from elastic2, started after elastic1. Elastic1 has the exact same log message, but with it's own IP:

[2019-06-12T13:13:47,739][INFO ][o.e.n.Node               ] [elastic2] starting ...
[2019-06-12T13:13:47,887][INFO ][o.e.t.TransportService   ] [elastic2] publish_address {10.192.10.62:9300}, bound_addresses {10.192.10.62:9300}
[2019-06-12T13:13:47,895][INFO ][o.e.b.BootstrapChecks    ] [elastic2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-12T13:13:47,903][INFO ][o.e.c.c.Coordinator      ] [elastic2] cluster UUID [gnWCd64oQlenU0d1eVaL6Q]
[2019-06-12T13:13:48,012][INFO ][o.e.c.s.MasterService    ] [elastic2] elected-as-master ([1] nodes joined)[{elastic2}{ML6K63J2Tvmx-WsPqeaeHA}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machine_memory=1
6819339264, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 45, version: 119, reason: master node changed {previous [], current [{elastic2}{ML6K63J2Tvmx-WsPqeaeHA
}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machine_memory=16819339264, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-06-12T13:13:48,072][INFO ][o.e.c.s.ClusterApplierService] [elastic2] master node changed {previous [], current [{elastic2}{ML6K63J2Tvmx-WsPqeaeHA}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machin
e_memory=16819339264, xpack.installed=true, ml.max_open_jobs=20}]}, term: 45, version: 119, reason: Publication{term=45, version=119}
[2019-06-12T13:13:48,176][INFO ][o.e.h.AbstractHttpServerTransport] [elastic2] publish_address {10.192.10.62:9200}, bound_addresses {10.192.10.62:9200}
[2019-06-12T13:13:48,176][INFO ][o.e.n.Node               ] [elastic2] started
[2019-06-12T13:13:48,401][INFO ][o.e.l.LicenseService     ] [elastic2] license [accb73b4-6b25-43ec-a3e4-a73575d5e69c] mode [basic] - valid
[2019-06-12T13:13:48,413][INFO ][o.e.g.GatewayService     ] [elastic2] recovered [0] indices into cluster_state

Brett_Blackham · June 12, 2019, 1:26pm

Here are the current configs.

DavidTurner · June 12, 2019, 1:32pm

I see different cluster UUIDs in two of the logs you shared, indicating you have formed different clusters. I think this note in the manual applies.

Brett_Blackham · June 12, 2019, 1:44pm

Mr Turner. I'm so happy I could kiss you. But we both might like that, which could be strange. So I'll just post my steps to get it working.

On all three servers I ran the following as root:

root@elastic1:~# sudo systemctl stop elasticsearch.service
root@elastic1:/etc/elasticsearch# cd /var/lib/elasticsearch/
root@elastic1:/var/lib/elasticsearch# ls
nodes
root@elastic1:/var/lib/elasticsearch# rm -r nodes

Then I "systemclt start elasticserach.service" on all three, but not before I had deleted the contents of path.data.

{
  "cluster_name" : "the-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Thank you again.

warkolm · June 12, 2019, 9:50pm

For future reference, please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them

system · July 10, 2019, 9:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Adding a second node to cluster Elasticsearch	11	2615	May 24, 2018
Add 2nd node to cluster Elasticsearch	3	450	July 5, 2020
Unable to add node to the Elasticsearch cluster Elasticsearch	4	327	August 25, 2021
Not able to add data node to a elasticsearch cluster Elasticsearch	4	866	November 20, 2018
HELP - 2nd node not joining cluster Elasticsearch	4	413	November 29, 2018

Difficulty adding nodes

Related topics