Cluster is not available and http request failed, but request by transport client worked well

[fanwang]

Elasticsearch version
V5.6.12

JVM version
JDK 1.8
OS version
CentOS7.6

Description of the problem including expected versus actual behavior :

1. In a cluster with 3 master nodes, minuimum_master_nodes is 2. Kill 2 of them, keep the cluster in a not-enough-master state.
2. Send the http request, such as, curl http://localhost:9200/_cat/nodes, and the request is not responsed, It's ok. But, if I send a request in this way, no exception or error was found.

TransportClient client = new PreBuiltTransportClient(Settings.EMPTY).addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300))

SearchResponse response = client.prepareSearch(index).setTypes(type).setSize(1000).setQuery(queryBuilder).execute().actionGet();

3. I expected an expection, but it worked well.

Any help will be appreciated， thanks.

[Christian_Dahlqvist]

It looks to me like the two requests are not identical. The search request should return even if no master is present while the other request requires access to information the master holds and will fail. Send the same request through the two clients and compare results.

[fanwang]

Are the two requests you mentioned sent by the same client, such as http client or transport client？It makes me confused, do you know how it happened? Thanks.

[DavidTurner]

You are running a very old version, long past the end of its supported life, and should upgrade as a matter of urgency.

This behaviour is what I'd expect from reading the documentation: by default if there's no master then searches work but admin requests don't. The difference in client is irrelevant.

edit sorry actually on closer inspection the docs suggest that admin requests would work in this state. I don't think that's true, at least not for something like GET _cat/nodes, although GET _cat/nodes/_local probably would.

[fanwang]

Thanks , I had to choose the version for some reasons. And, I took a inspection at the docs, the setting, discovery.zen.no_master_block, is the key point. But I thought there may be some wrong descriptions in the docs.

The discovery.zen.no_master_block setting has two valid options:
all
All operations on the node—​i.e. both read & writes—​will be rejected. This also applies for api cluster state read or write operations, like the get index settings, put mapping and cluster state api.
(default)
Write operations will be rejected. Read operations will succeed, based on the last known cluster configuration. This may result in partial reads of stale data as this node may be isolated from the rest of the cluster.

According to my test, all of these read requests were rejected in the state. Such as,
POST index/type -d '{"query": {"match_all"：{}}}'
GET _cat/nodes
GET _cat/nodes/_local

Did I ignore something important?