# How to determine if one of master gone from cluster?
**URL:** https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171
**Category:** Elasticsearch
**Created:** [July 14, 2020, 5:25pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171 "2020-07-14T17:25:33Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@byumov](https://discuss.elastic.co/u/byumov)
#### Post date: [July 14, 2020, 5:25pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/1 "2020-07-14T17:25:33Z")
Hi! I have a clean and simple installation os ES:
9 nodes: 6 data and 3 masters(without data).
I want to determine situation when one of masters gone for some reason(power off).
How I can do it on 7 ES? I looked at` /_cluster/health` and `/_nodes` endpoints and documentation, but failed to found answers.
Endpoints above show only available nodes and didn't show powered off nodes.
On previous version os ES(6) I could do it easily:
1. Get minimum\_master\_nodes property from `/_settings`
2. Get all nodes, count masters, and compare them with minimum\_master\_nodes.
If count of masters wich I get from API \<= minimum\_master\_nodes, so, some master nodes down.
From version 7 there are no options `discovery.zen.minimum_master_nodes`.
How I can do something similar?
P.S. It would be great to do this check without hardcode master nodes(or their count) somewhere. And sorry for my English 🙂
Thank you!
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 14, 2020, 11:33pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/2 "2020-07-14T23:33:30Z")
You could use `_cat/nodes`, this shows what roles each node is set to and which is the master.
But you'd need to maintain the state of what that _should_ look like yourself, as the APIs do not do that.
What's the driver behind wanting this info?
---
### Author:  [@byumov](https://discuss.elastic.co/u/byumov)
#### Post date: [July 15, 2020, 1:03pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/3 "2020-07-15T13:03:43Z")
@warkolm, thank you for answer!
I'll try to explain our case.
We use the cloud in production (for simplicity we can assume that these are Kubernetes).
There is a special balancing mechanism in the cloud that can move instances (containers) between HW servers (even if it is stateful).
Our goal is to prevent a situation where the cloud decides to transfer two of the three masters at once simultaneously.
Each instance report to the cloud about itself(making some health check locally). The cloud looks at report and makes a decision: can an instance be stopped or moved. To do this, each instance must have information about the state of the cluster (about the availability of other masters).
For example, master1 must know, about availability of master2 and master3, to allow the cloud to stop or move it.
Without this information, we can get a situation where two from three master instances stopped by the cloud and we lose ES cluster.
Unfortunately, `_/cluster/health` and other endpoint do not give any information if one of master gone.
So, the goal: each node must know current cluster state.
Yes, we can use some external services like consul or hardcoded cluster nodes, but it would be great if ES cluster could report about his master nodes state(if I understand correctly, each master already store information about other masters, but that information unavailable from API)
Sorry for the long and possibly confusing explanation.
---
### Author:  [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)
#### Post date: [July 15, 2020, 1:38pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/4 "2020-07-15T13:38:58Z")
If you are using K8s have you looked at pod disruption budget (PDB)?
---
### Author:  [@byumov](https://discuss.elastic.co/u/byumov)
#### Post date: [July 15, 2020, 2:05pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/5 "2020-07-15T14:05:31Z")
@Vinayak_Sapre, hi! No, it's not Kubernetes, but, in Kubernetes terminology, we have three total independent Kubernetes clusters. One of three ES master in each cluster. So, we have the same problem - one Kubernete cluster doesn't know anything about other clusters and applications. But application(ES) know about itself.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [July 15, 2020, 2:59pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/6 "2020-07-15T14:59:12Z")
can you check cluster stat? if it is green then you are good to move.
anything else you can't move (this will be only true for three node cluster)
if you have five node cluster then two node can go out and cluster stat is still green.
GET /\_cat/health?pretty
---
### Author:  [@byumov](https://discuss.elastic.co/u/byumov)
#### Post date: [July 15, 2020, 3:19pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/7 "2020-07-15T15:19:21Z")
@elasticforme, hi! Thank you for advice.
But, from documentation: `green means that all shards are allocated`.
[https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)
So, if we have 3 dedicated master(without data) when one of masters goes down, `/_cat/health` will show green status.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [July 15, 2020, 3:32pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/8 "2020-07-15T15:32:57Z")
can you do some hard check. for example if you have three node
and GET /\_cat/health returns number\_of\_nodes less then 3 don't move.
this might not be best option though.
---
### Author:  [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [July 15, 2020, 5:39pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/9 "2020-07-15T17:39:21Z")
So if I state in another way, you want to be alerted when one of your master is not in the cluster anymore. Since shards are only sitting on data nodes, health will still show as green.
Any other constraints you want to highlight here ?
---
### Author:  [@byumov](https://discuss.elastic.co/u/byumov)
#### Post date: [July 15, 2020, 6:17pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/10 "2020-07-15T18:17:38Z")
@Ayush_Mathur, yes, you understood correctly. And no other constraints.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [July 15, 2020, 7:59pm UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/11 "2020-07-15T19:59:54Z")
> [@elasticforme](#):
>
> mber\_
another option is
GET /\_nodes/\_all,master:true
and if failed is \> 0 then don't move
"\_nodes" : {
"total" : 3,
"successful" : 3,
"failed" : 0
},
---
### Author:  [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)
#### Post date: [July 16, 2020, 6:44am UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/12 "2020-07-16T06:44:16Z")
That is a good way, though I wonder when a master will drop from that list, i.e. no longer be failed.
Also can do like this and count the nodes with the 'm' in their role and compare to your goal of 3; if only 2; you've lost a master.
GET /\_cat/nodes?h=name,node.role
---
### Author:  [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [July 27, 2020, 9:08am UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/13 "2020-07-27T09:08:35Z")
You can just put up a watcher with http(s) input and make a call to \_nodes/master:true endpoint. In condition, check if `ctx.payload._nodes.total < expected_masters`, return true and trigger action of your like - mail, slack, add doc in alerts index, etc. etc.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 24, 2020, 9:08am UTC](https://discuss.elastic.co/t/how-to-determine-if-one-of-master-gone-from-cluster/241171/14 "2020-08-24T09:08:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.