Not enough master nodes/failed to retrieve user

[pan_e]

I'm using ElasticSearch 6.8 with 3 Nodes. I have set discovery.zen.minimum_master_nodes: 2

I'm trying to test what happens when there is only 1 node online.
I've shut down 2 nodes and I can see the 3rd node goes into: "not enough master nodes discovered during pinging"

Based on: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-discovery-zen.html#no-master-block I would expect to still be able to run searches against this node.

This seems to work fine as the Elastic user but if I try as an custom user I get an exception suggesting that the server is trying to connect to another node:
[2021-01-14T15:48:16,669][DEBUG][o.e.a.g.TransportGetAction] [SERVER-A] null: failed to execute [get [.security][doc][user-testuser]: routing [null]]
org.elasticsearch.transport.NodeNotConnectedException: [SERVER-B][192.168.1.10:9300] Node not connected

This suggests that somehow the authentication is depending on the other node.

All nodes are set to:

node.master: true
node.data: true
node.max_local_storage_nodes: 1
node.attr.fault_domain: 0
node.attr.update_domain: 0
cluster.routing.allocation.awareness.attributes: fault_domain,update_domain

The user basically has full access to any indexes with a prefix:

`      ` `"cluster"` `: [` `"all"` `],`

`      ` `"indices"` `: [        {`

`          ` `"names"` `: [`

`            ` `"$($IndexName)_*"`

`          ` `],`

`          ` `"privileges"` `: [`

`            ` `"all"`

`          ` `],`

`          ` `"field_security"` `: {`

`            ` `"grant"` `: [`

`              ` `"*"`

`            ` `]`

`          ` `},`

`          ` `"allow_restricted_indices"` `: false`

`        ` `}`

`      ` `],`

`      ` `"applications"` `: [ ],`

`      ` `"run_as"` `: [ ],`

`      ` `"metadata"` `: { },`

`      ` `"transient_metadata"` `: { ` `"enabled"` `: true }`

`    ` `}`

I'm not sure what could be wrong/why searches are not working?

[warkolm (Mark Walkom)]

Welcome to our community!

Does the node that is left in the cluster have any of the .security* shards on it at all?

[pan_e]

thank you.

indeed the shard is missing from that node. it seems to be related to the dynamic values set for auto_expand_replicas. looking into why that may be.

looking at security-index-template that has "auto_expand_replicas : 0-1", I wonder why that is.

[pan_e]

found this which is slightly concerning:

Scale in a cluster - Elasticsearch - Alibaba Cloud Documentation Center

What do I do if data nodes fail to be removed or data fails to be migrated due to the auto_expand_replicas index setting?
Cause: Some users may use the access control feature provided by the X-Pack plug-in. In earlier Elasticsearch versions, this feature applies the "index.auto_expand_replicas" : "0-all" setting to the .security index by default. This causes errors when you migrate data or remove data nodes.