Nest and a cluster without a master node

[aslamy1 (Aslamy)]

I have a cluster with two nodes.
I use discovery.zen.minimum_master_nodes : 2 to avoid the splitt brain.

When I stop one of this nodes. Nest stops working. {" Invalid NEST response built from a unsuccessful low level call on POST: "}, Failed to ping the specified node.

But when I run http://x.x.x.x:9200/index/_search on internet browser I get 200 Ok response.

[JKhondhu (Jymit Singh Khondhu)]

What node is your NEST code pointing to?
Are both nodes, master and data?

[aslamy1 (Aslamy)]

Hi.
Yes It points to both Master and data node

[JKhondhu (Jymit Singh Khondhu)]

@JKhondhu

Are both nodes, master and data?

...?

[aslamy1 (Aslamy)]

Yes both are master and data nodes:

  	"roles": [
  		"master",
  		"data",
  		"ingest"
  	],

[forloop (Russ Cam)]

Can you show how you are constructing the client, and what type of IConnectionPool you are using.

The audit trail on a failed response will also be useful to understand what the client does when one of the nodes is stopped. This information is available on the .DebugInformation on the response

[forloop (Russ Cam)]

@aslamy1

I have a cluster with two nodes.

I use discovery.zen.minimum_master_nodes : 2 to avoid the splitt brain.

With this configuration, when a node goes down, the entire cluster will be inoperable, since a minimum of 2 master eligible nodes are needed in the cluster. I would recommend adding another master eligible node to avoid the cluster becoming inoperable when a node goes down.

[aslamy1 (Aslamy)]

With this configuration, when a node goes down, the entire cluster will be inoperable, since a minimum of 2 master eligible nodes are needed in the cluster. I would recommend adding another master eligible node to avoid the cluster becoming inoperable when a node goes down.

I know this scenario is bad. But my question is why can I read data from the other alive node if I use raw queries, but Nest will fail?

In this scenario when there are no master node I want simply just read data from the other node. (I know in this situation create/delete/update is bad operations to do)

Can you show how you are constructing the client, and what type of IConnectionPool you are using.

     var nodes = new List<Uri> {new Uri("http://x.x.x.125:9200"), new Uri("http://x.x.x.126:9200") };
     var pool = new StaticConnectionPool(nodes);
     var connectionSettings = new ConnectionSettings(pool)
                .DefaultTypeNameInferrer(d => d.Name);
     var elasticClient = new ElasticClient(connectionSettings);

I also tried with:
SniffingConnectionPool
StickyConnectionPool
but the result was the same.

The audit trail on a failed response will also be useful to understand what the client does when one of the nodes is stopped. This information is available on the .DebugInformation on the response 1

I have two nodes, http://x.x.x.125:9200 and http://x.x.x.126:9200 .
In my test I stoped node http://x.x.x.125:9200 but node http://x.x.x.126:9200 still running.

DebugInformation:

Invalid NEST response built from a unsuccessful low level call on POST: /index/document/_search?pretty=true&typed_keys=true

Audit trail of this API call:

• [1] AllNodesDead: Took: 00:00:00
• [2] Resurrection: Node: http://x.x.x.125:9200/ Took: 00:00:00
• [3] PingFailure: Node: http://x.x.x.125:9200/ Exception: PipelineException Took: 00:00:01.0088607
• [4] MaxRetriesReached: Took: -736804.13:39:12.3547679

OriginalException: Elasticsearch.Net.ElasticsearchClientException: Maximum number of retries reached. ---> System.AggregateException: One or more errors occurred. ---> Elasticsearch.Net.PipelineException: Failed to ping the specified node. ---> Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the specified node.

vid Elasticsearch.Net.RequestPipeline.Ping(Node node)
--- Slut på stackspårning för interna undantag ---
vid Elasticsearch.Net.RequestPipeline.Ping(Node node)
vid Elasticsearch.Net.Transport1.Ping(IRequestPipeline pipeline, Node node) vid Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- Slut på stackspårning för interna undantag ---
--- Slut på stackspårning för interna undantag ---

Audit exception in step 3 PingFailure:

Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the specified node.
vid Elasticsearch.Net.RequestPipeline.Ping(Node node)

Request:

<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

Response:

[aslamy1 (Aslamy)]

I found this in documentation:

No master block
For the cluster to be fully operational, it must have an active master and the number of running master eligible nodes must satisfy the discovery.zen.minimum_master_nodes setting if set. The discovery.zen.no_master_block settings controls what operations should be rejected when there is no active master.

The discovery.zen.no_master_block setting has two valid options:

all

All operations on the node—i.e. both read & writes—will be rejected. This also applies for api cluster state read or write operations, like the get index settings, put mapping and cluster state api.

write

(default) Write operations will be rejected. Read operations will succeed, based on the last known cluster configuration. This may result in partial reads of stale data as this node may be isolated from the rest of the cluster.

So I want simply read data when there is no active master. But this fails on nest.

[aslamy1 (Aslamy)]

I found a potential solution.
When I use DisablePing then it works. But some request fails (Reqests which reaches dead nod) and no search response returns.

var pool = new StaticConnectionPool(nodes);
var connectionSettings = new ConnectionSettings(pool)
.DisablePing()

Is it a bug or it should work in this way ?