Difficulty adding nodes

Elastic Stack Elasticsearch
1

I've been following several tutorials/examples but I am not able to get elastic to see other nodes.

I have 3 VMs running Ubuntu 18.04 LTS

/etc/hosts
10.192.10.61 elastic1
10.192.10.62 elastic2
10.192.10.63 elastic3

I can telnet to port 9200 and 9300 on all three servers. But I can never get it to show more than one node. I imagine I'm missing something silly in my config.

root@elastic1:/var/log# curl -XGET 'http://10.192.10.61:9200/_cluster/health?pretty'
{
  "cluster_name" : "the-cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : "NaN"
}

Here are the config files
image
image
image

Tail of the cluster log
[2019-06-12T00:03:06,983][INFO ][o.e.p.PluginsService ] [elastic1] loaded module [x-pack-security]
[2019-06-12T00:03:06,984][INFO ][o.e.p.PluginsService ] [elastic1] loaded module [x-pack-sql]
[2019-06-12T00:03:06,984][INFO ][o.e.p.PluginsService ] [elastic1] loaded module [x-pack-watcher]
[2019-06-12T00:03:06,985][INFO ][o.e.p.PluginsService ] [elastic1] no plugins loaded
[2019-06-12T00:03:11,566][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic1] [controller/5736] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV
[2019-06-12T00:03:12,369][INFO ][o.e.d.DiscoveryModule ] [elastic1] using discovery type [zen] and seed hosts providers [settings]
[2019-06-12T00:03:13,324][INFO ][o.e.n.Node ] [elastic1] initialized
[2019-06-12T00:03:13,325][INFO ][o.e.n.Node ] [elastic1] starting ...
[2019-06-12T00:03:13,490][INFO ][o.e.t.TransportService ] [elastic1] publish_address {10.192.10.61:9300}, bound_addresses {10.192.10.61:9300}
[2019-06-12T00:03:13,499][INFO ][o.e.b.BootstrapChecks ] [elastic1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-12T00:03:13,507][INFO ][o.e.c.c.Coordinator ] [elastic1] cluster UUID [8V5kNevdQDuGgqMCCjRg5A]
[2019-06-12T00:03:13,667][INFO ][o.e.c.s.MasterService ] [elastic1] elected-as-master ([1] nodes joined)[{elastic1}{DpJg-1UISCSBF_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine_memory=16819339264, xpack.installed=true, ml.max_ope
n_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 37, version: 95, reason: master node changed {previous , current [{elastic1}{DpJg-1UISCSBF_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine_memory=16819339264,
xpack.installed=true, ml.max_open_jobs=20}]}
[2019-06-12T00:03:13,729][INFO ][o.e.c.s.ClusterApplierService] [elastic1] master node changed {previous , current [{elastic1}{DpJg-1UISCSBF_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine_memory=16819339264, xpack.installed=true,
ml.max_open_jobs=20}]}, term: 37, version: 95, reason: Publication{term=37, version=95}
[2019-06-12T00:03:13,806][INFO ][o.e.h.AbstractHttpServerTransport] [elastic1] publish_address {10.192.10.61:9200}, bound_addresses {10.192.10.61:9200}
[2019-06-12T00:03:13,806][INFO ][o.e.n.Node ] [elastic1] started
[2019-06-12T00:03:45,840][INFO ][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][young][32][3] duration [778ms], collections [1]/[1.4s], total [778ms]/[844ms], memory [582.4mb]->[212.8mb]/[1.9gb], all_pools {[young] [527.1mb]->[19.5mb]/[532.5mb]}{[survivor] [55.3mb]
->[56.1mb]/[66.5mb]}{[old] [0b]->[138.9mb]/[1.3gb]}
[2019-06-12T00:03:45,841][WARN ][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][32] overhead, spent [778ms] collecting in the last [1.4s]

Can anyone see what I'm missing?

2

Hi,

I guess you need to configure "cluster.initial_master_nodes" in your elasticsearch.yml.

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html

Thanks.

1 Like
3

Thanks for looking. I have tried it with cluster.initial_master_nodes commented out. Also with just elastict1, and with elastic2 and elastic3. Same results every time.

The only setting that seems to change the _cluster/health is gateway.recover_after_nodes. Setting it to 2 will start but with a "status": "red". A value of 1 (or commented out) will start with a "status": "green".

I found someone else that was getting this same issue. In fact, I recognize his configs. He used one of the 5 different example/tutorials I saw. So, I'm not the only one running into this.
He had the same, or similar, issue

Hari.v said he got his cluster to work by adding "node.master: true , node.data: true for all the 3 nodes". I modified my config to mach his (again), but now I have three ES servers each running as individual (single node) clusters.

Below is the log file from elastic2, started after elastic1. Elastic1 has the exact same log message, but with it's own IP:

[2019-06-12T13:13:47,739][INFO ][o.e.n.Node               ] [elastic2] starting ...
[2019-06-12T13:13:47,887][INFO ][o.e.t.TransportService   ] [elastic2] publish_address {10.192.10.62:9300}, bound_addresses {10.192.10.62:9300}
[2019-06-12T13:13:47,895][INFO ][o.e.b.BootstrapChecks    ] [elastic2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-12T13:13:47,903][INFO ][o.e.c.c.Coordinator      ] [elastic2] cluster UUID [gnWCd64oQlenU0d1eVaL6Q]
[2019-06-12T13:13:48,012][INFO ][o.e.c.s.MasterService    ] [elastic2] elected-as-master ([1] nodes joined)[{elastic2}{ML6K63J2Tvmx-WsPqeaeHA}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machine_memory=1
6819339264, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 45, version: 119, reason: master node changed {previous [], current [{elastic2}{ML6K63J2Tvmx-WsPqeaeHA
}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machine_memory=16819339264, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-06-12T13:13:48,072][INFO ][o.e.c.s.ClusterApplierService] [elastic2] master node changed {previous [], current [{elastic2}{ML6K63J2Tvmx-WsPqeaeHA}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machin
e_memory=16819339264, xpack.installed=true, ml.max_open_jobs=20}]}, term: 45, version: 119, reason: Publication{term=45, version=119}
[2019-06-12T13:13:48,176][INFO ][o.e.h.AbstractHttpServerTransport] [elastic2] publish_address {10.192.10.62:9200}, bound_addresses {10.192.10.62:9200}
[2019-06-12T13:13:48,176][INFO ][o.e.n.Node               ] [elastic2] started
[2019-06-12T13:13:48,401][INFO ][o.e.l.LicenseService     ] [elastic2] license [accb73b4-6b25-43ec-a3e4-a73575d5e69c] mode [basic] - valid
[2019-06-12T13:13:48,413][INFO ][o.e.g.GatewayService     ] [elastic2] recovered [0] indices into cluster_state
4

Here are the current configs.

image

image

image

5

I see different cluster UUIDs in two of the logs you shared, indicating you have formed different clusters. I think this note in the manual applies.

1 Like
6

Mr Turner. I'm so happy I could kiss you. But we both might like that, which could be strange. So I'll just post my steps to get it working.

On all three servers I ran the following as root:

root@elastic1:~# sudo systemctl stop elasticsearch.service
root@elastic1:/etc/elasticsearch# cd /var/lib/elasticsearch/
root@elastic1:/var/lib/elasticsearch# ls
nodes
root@elastic1:/var/lib/elasticsearch# rm -r nodes

Then I "systemclt start elasticserach.service" on all three, but not before I had deleted the contents of path.data.

{
  "cluster_name" : "the-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Thank you again.

1 Like
7

For future reference, please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them :slight_smile:

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.