Have upgraded production to 7.7.1 oss
Have created brand new node also running 7.7.1 oss
Still no luck. After 3 days I'm getting close to throwing ES out.
The new node sees the 'master'. It joins the cluster. Now the issue is that no data is replicating.
Cluster status is red.
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "zm-amz-data",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 29,
"active_shards" : 29,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 39,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 42.64705882352941
}
There are no errors on the new node or the master.
[2020-06-07T00:07:00,644][INFO ][o.e.c.s.MasterService ] [elasticsearch-01] node-join[{elasticsearch-02}{e_toEmodToGU98qY6MZaWQ}{kiBIItOtRpql0GcThMtkHg}{<new node ip>}{<new node ip>:9300}{dimr} join existing leader], term: 3, version: 154, delta: added {{elasticsearch-02}{e_toEmodToGU98qY6MZaWQ}{kiBIItOtRpql0GcThMtkHg}{<new node ip>}{<new node ip>:9300}{dimr}}
[2020-06-07T00:07:01,131][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-01] added {{elasticsearch-02}{e_toEmodToGU98qY6MZaWQ}{kiBIItOtRpql0GcThMtkHg}{<new node ip>}{<new node ip>:9300}{dimr}}, term: 3, version: 154, reason: Publication{term=3, version=154}
All I want is to get all our data from the master onto the new node! I am amazed how difficult this seems to be.