Add 2nd node to cluster

Elastic Stack Elasticsearch
1

We've been running a production system with a single node for over a year and decided to bump up to having 2 nodes for some resiliency.
I cannot for the life of me figure out how to get this new node to see the 'master' and join it.

I can telnet between machines without issue.
I can issue curl commands between machines.
I can 'reindex' from the master to a new machine without issue.

The only possible hiccup could be that the new node machine is running 7.7.0 and the master is on 6.8.9.

on both machines:

cluster.name: xyz-es-cluster
node.name: ${HOSTNAME}

on new node:

discovery.seed_hosts: ["10.xxx.xx.xxx"] # master ip
cluster.initial_master_nodes: ["hostname of master"]

on master node:
discovery.zen.ping.unicast.hosts: ["10.xxx.xx.xxx", "10.xxx.xx.xxx"] #master node ip, new cluster node ip. in this format because it is 6.8.9

Otherwise, the elasticsearch.yml's on both are as installed by the package manager.

is there something else that needs doing?

2

Both nodes need to have use the same version.

4

Have upgraded production to 7.7.1 oss
Have created brand new node also running 7.7.1 oss
Still no luck. After 3 days I'm getting close to throwing ES out.
The new node sees the 'master'. It joins the cluster. Now the issue is that no data is replicating.
Cluster status is red.

curl -XGET 'http://localhost:9200/_cluster/health?pretty'
{
  "cluster_name" : "zm-amz-data",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 29,
  "active_shards" : 29,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 39,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 42.64705882352941
}

There are no errors on the new node or the master.

[2020-06-07T00:07:00,644][INFO ][o.e.c.s.MasterService    ] [elasticsearch-01] node-join[{elasticsearch-02}{e_toEmodToGU98qY6MZaWQ}{kiBIItOtRpql0GcThMtkHg}{<new node ip>}{<new node ip>:9300}{dimr} join existing leader], term: 3, version: 154, delta: added {{elasticsearch-02}{e_toEmodToGU98qY6MZaWQ}{kiBIItOtRpql0GcThMtkHg}{<new node ip>}{<new node ip>:9300}{dimr}}
[2020-06-07T00:07:01,131][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-01] added {{elasticsearch-02}{e_toEmodToGU98qY6MZaWQ}{kiBIItOtRpql0GcThMtkHg}{<new node ip>}{<new node ip>:9300}{dimr}}, term: 3, version: 154, reason: Publication{term=3, version=154}

All I want is to get all our data from the master onto the new node! I am amazed how difficult this seems to be.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.