# Not enough master nodes/failed to retrieve user
**URL:** https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165
**Category:** Elasticsearch
**Tags:** elastic-stack-security
**Created:** [January 14, 2021, 6:16pm UTC](https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165 "2021-01-14T18:16:03Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@pan\_e](https://discuss.elastic.co/u/pan_e)
#### Post date: [January 14, 2021, 6:16pm UTC](https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165/1 "2021-01-14T18:16:03Z")
I'm using ElasticSearch 6.8 with 3 Nodes. I have set discovery.zen.minimum\_master\_nodes: 2
I'm trying to test what happens when there is only 1 node online.
I've shut down 2 nodes and I can see the 3rd node goes into: "not enough master nodes discovered during pinging"
Based on: [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-discovery-zen.html#no-master-block](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-discovery-zen.html#no-master-block) I would expect to still be able to run searches against this node.
This seems to work fine as the Elastic user but if I try as an custom user I get an exception suggesting that the server is trying to connect to another node:
[2021-01-14T15:48:16,669][DEBUG][o.e.a.g.TransportGetAction] [SERVER-A] null: failed to execute [get [.security][doc][user-testuser]: routing [null]]
org.elasticsearch.transport.NodeNotConnectedException: [SERVER-B][192.168.1.10:9300] Node not connected
This suggests that somehow the authentication is depending on the other node.
All nodes are set to:
```auto
node.master: true
node.data: true
node.max_local_storage_nodes: 1
node.attr.fault_domain: 0
node.attr.update_domain: 0
cluster.routing.allocation.awareness.attributes: fault_domain,update_domain
```
The user basically has full access to any indexes with a prefix:
```auto
` ` `"cluster"` `: [` `"all"` `],`
` ` `"indices"` `: [ {`
` ` `"names"` `: [`
` ` `"$($IndexName)_*"`
` ` `],`
` ` `"privileges"` `: [`
` ` `"all"`
` ` `],`
` ` `"field_security"` `: {`
` ` `"grant"` `: [`
` ` `"*"`
` ` `]`
` ` `},`
` ` `"allow_restricted_indices"` `: false`
` ` `}`
` ` `],`
` ` `"applications"` `: [],`
` ` `"run_as"` `: [],`
` ` `"metadata"` `: { },`
` ` `"transient_metadata"` `: { ` `"enabled"` `: true }`
` ` `}`
```
I'm not sure what could be wrong/why searches are not working?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 14, 2021, 11:53pm UTC](https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165/2 "2021-01-14T23:53:45Z")
Welcome to our community! 😃
Does the node that is left in the cluster have any of the `.security*` shards on it at all?
---
### Author:  [@pan\_e](https://discuss.elastic.co/u/pan_e)
#### Post date: [January 15, 2021, 12:11pm UTC](https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165/3 "2021-01-15T12:11:27Z")
thank you.
indeed the shard is missing from that node. it seems to be related to the dynamic values set for auto\_expand\_replicas. looking into why that may be.
looking at security-index-template that has "auto\_expand\_replicas : 0-1", I wonder why that is.
---
### Author:  [@pan\_e](https://discuss.elastic.co/u/pan_e)
#### Post date: [January 15, 2021, 2:55pm UTC](https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165/4 "2021-01-15T14:55:46Z")
found this which is slightly concerning:
> **[Scale in a cluster - Elasticsearch - Alibaba Cloud Documentation Center](https://www.alibabacloud.com/help/en/es/user-guide/scale-in-a-cluster)**
>
> Scale in a cluster,Elasticsearch:If your business is in the off-peak hours of traffic or the volume of data stored in your cluster decreases, you can remove data nodes from your cluster to scale in the cluster. This topic describes h...
> What do I do if data nodes fail to be removed or data fails to be migrated due to the `auto_expand_replicas` index setting?
> Cause: Some users may use the access control feature provided by the X-Pack plug-in. In earlier Elasticsearch versions, this feature applies the `"index.auto_expand_replicas" : "0-all"` setting to the .security index by default. This causes errors when you migrate data or remove data nodes.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 12, 2021, 2:56pm UTC](https://discuss.elastic.co/t/not-enough-master-nodes-failed-to-retrieve-user/261165/5 "2021-02-12T14:56:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.