Elasticsearch - Changing Master-eligible nodes to dedicated Master nodes

Elastic Stack Elasticsearch
1

Hi All,

We have a Elasticsearch 8.12.2 cluster that was very recently set up. This cluster has 3 "Master Eligible" and 5 "Data" nodes.

When I run curl command to check cluster health it shows it as green with 8 data nodes.

I need to change the Master eligible to "Dedicated Master" nodes.

I tried shutting down the entire cluster and repurpose the Master eligble nodes. It gets re-purposed fine but for some reason the cluster health becomes red and the whole set up goes south.

I am yet to configure Kibana and other stuff.

Please help.

[ptal@elasticsearch]$ curl -k -u elastic:XXXXXXXX https://`hostname`:44444/_cluster/health?pretty
{
  "cluster_name" : "elkcluster-prod",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 8,
  "number_of_data_nodes" : 8,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
2

Do you have any data on the cluster that you need to keep or you can rebuild it?

It seems that your cluster is basically empty, it shows only 2 shards.

If you can rebuild it from scratch, just stop everything, delete the data folder, and then configure the master eligible nodes to have only the master role, then start your nodes.

3

Thanks.

Just built this cluster today; so barely anything on it.

Shall I delete data folder from "all" the nodes or just the presently "Master eligible" ones?

4

If you can start the cluster from scratch then you need to delete everything, from all nodes.

This will bootstrap a fresh new cluster, just make sure to configure the master nodes as master only.

5

I plan to do the following. Hope it works :blush:

  1. Shutdown the entire cluster (8 nodes)

  2. Remove/ move data directory from all nodes

  3. Update elasticsearch.yml on master nodes to reflect node.role as “Master”

  4. Bring up master nodes and check log for election of a master node.

  5. Bring up data nodes

6

This needs to be done at the same time as you need data nodes for the cluster to work.

So you will need to start the master nodes and the data nodes and wait for the cluster to form.