How exactly do I shutdown, then restart the cluster?

Elastic Stack Elasticsearch
1

I have a cluster of 3 nodes. I need to change a setting in elasticsearch.yml and then restart the cluster. I am a bit confused on the procedure. Here is what I have so far. Is this correct?

  1. Make changes to /etc/elasticsearch/elasticsearch.yml on all 3 nodes in the cluster.
  2. On the master node only, run sudo service elasticsearch restart

Is this all there is to it? Or do I need to also run commands on the child nodes as well?

2

If you have 3 master eligible nodes you can perform a rolling restart without any outage because stopping one node at the time you will keep the quorum (2 out of 3).

The procedures are detailed at https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-rolling

1 Like
3

@Luca_Belluccini Sorry for a dumb question, how do I know if the nodes are master eligible?

And also, if I don't care about causing an outage - is the procedure I outlined above sufficient?

4

There is two way you can do.
shutdown whole cluster and restart, if you can afford stopping ingestion

execute this on all three nodes
systemctl stop kibana
systemctl stop logstash

then stop elasticsearch simultaniously
systemctl stop elasticsearch

and then start in reverse.

otherwise one node at a time(rolling update). but on this procedure don't forget to disable shard reallocation for faster restaart. make sure your cluster is green before you stop/start second node.

1 Like
5

check on your /etc/elasticsearch/elasticsearch.yml file
node.master: true that means it is master node

1 Like
6

The link I've shared details also the full cluster restart procedure (at the top of the page).

To know if a node is master or not, check the API GET _cat/nodes?v and check for the column r or node.role. More info at https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodes.html

By default, all nodes are master eligible.

1 Like
7

So all three nodes report that they are master in elasticsearch.yml

node.master: true
node.data: true

However, when I run GET /_cat/nodes, I get this:

10.147.36.11 48 99 1 6.64 6.74 6.35 mdi - es-oc2-q01
10.147.36.13 60 99 1 4.59 4.60 4.15 mdi * es-oc2-q03
10.147.36.12 54 99 3 7.29 8.33 7.84 mdi - es-oc2-q02.rediv.int

I assumed that the presence of an asterisk * indicated that it's the master. Is that not the case?
Or does it indicate that all nodes are master-eligible and have chosen es-oc2-q03 node to be master via the quorum?

Does it change anything with regards to the running the commands you outlined? Meaning, the order of the nodes on which I run them?

8

Yes you have 3 master eligible nodes and one of them is master elected at the moment (the one with *).

The best 2 ways to restart the cluster are the ones linked in the documentation.

1 Like
9

m=master

all nodes are master nodes. but q03 is taking lead. hence if you shutdown that automatically another will take lead. and that will have *.

process is completely random.

I had three node cluster for long time and I did upgrade/update by shutting down everything and starting all.
order is does not matter here because you shutting down all services.

now I have larger cluster and I now do rolling upgrade/update it takes little longer that is it.
In this process as well no order matter.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.