Primary Shards Unassigned but still Cluster State health Yellow

Elastic Stack Elasticsearch
1

I was testing by Creating two dedicated master node and one data node. So I put down my data node from the Cluster and then tried to create new Index by curl -X PUT "localhost:9200/customer?pretty"
where customer is the name of my index. The response of this was
{
"acknowledged" : true,
"shards_acknowledged" : false,
"index" : "customer"
}

So the shards are unassigned. Even the primary shards are unassigned. And when I do _cat/indices

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   customer 4te5lNcERcOwNiQjcRxBUA   5   1

The health of the Index is Yellow and the Cluster state is even yellow.
Why is that unexpected behavior since primary shards are even unassigned . (_cat/shards response)
customer 3 p UNASSIGNED
customer 3 r UNASSIGNED
customer 4 p UNASSIGNED
customer 4 r UNASSIGNED
customer 2 p UNASSIGNED
customer 2 r UNASSIGNED
customer 1 p UNASSIGNED
customer 1 r UNASSIGNED
customer 0 p UNASSIGNED
customer 0 r UNASSIGNED

Why is this unusual behavior since primary are unassigned but still Cluster State is Yellow?

2

A newly-created index doesn't immediately cause the cluster health to be red:

3

The documentation mentions that if the Index is still in INITIALIZING state then the Cluster State is Yellow. But in my scenario the Index State is UNASSIGNED.

/**
* The shard is unassigned (not allocated to any node).
*/
public boolean unassigned() {
return state == ShardRoutingState.UNASSIGNED;
}

4

This is true, but doesn't seem to be relevant. If a primary is UNASSIGNED then the logic I shared applies, allowing the health to report as yellow under some circumstances.

5

So after what time the Cluster state will be RED? Supposing that data node never recovers.

6

The cluster will go red if the shards can definitely not be allocated anywhere:

It will also go red if they are allocated and then this allocation fails:

If the shard is not successfully allocated then eventually one of these cases will happen.

7

So the Elasticsearch will internally try to make attempts to allocate as per setting. (which is 5 by default) index.allocation.max_retries . Then it will turn RED. Or one needs to do a reroute and if that reroute fails then it will turn RED?

8

No, the health will turn RED if any attempt fails, even if it will still retry:

unassignedInfo.getNumFailedAllocations() == 0
9

I waited for a long time to turn RED. But it never turn RED. So is it not trying allocation again until it reaches index.allocation.max_retries . But as I know ElasticSearch internally tries to allocate primary shards till this count. Suppose it had tried and each try it didnt find the data node, eventually it should be turning RED. But is not. I havent changed any settings in the elasticsearch.yml file except the setting for dedicated Master Node. Can u pls tell why is this behaviour?
(Just a beginner in ElasticSearch. Pls keep patience.)

10

I see, this wasn't clear from your initial post. It sounds like you're interested in why these shards are not allocated, not why the health is yellow instead of red. The best way to answer that is with the allocation explain API:

GET /_cluster/allocation/explain
{
  "index": "customer",
  "shard": 0,
  "primary": true
}
11

I used this allocations explain API and as expected it told the same reason .( as it is not having any data node.) I am actually more interested in turning it RED deliberately . I want to know when will this turn into RED as it is currently yellow. And if it internally tries to make attempts for allocation until index.allocation.max_retries count. And if this count eventually will exceed so why is this not turning RED? as it finally fails to allocate this shard on data node(and i dont have any data node)

12

Please could you share the full output of the allocation explain API? This sounds unexpected, but it's hard to be sure without seeing all the details.

13

{
"index" : "customer",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "INDEX_CREATED",
"at" : "2019-04-10T10:29:55.937Z",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes"
}

As I am not having any data node so it is unable to allocate in any node. This is understood. But since Elasticsearch internally tries to allocate till index.allocation.max_retries count. And eventually this count will exceed . and after it exceeds the status should turn RED. but is not. It is still yellow. Can u pls where is the issue? As I want to see the state as RED

14

Ok, I see. We don't even bother trying to assign any shards if there aren't any data nodes in the cluster, so the assignment never fails and hence the health never goes RED:

I suppose you could consider this a bug, but it's only possible to trigger this in very strange circumstances. Note that a cluster with no data nodes would already have RED health if you'd created any indices before shutting the last data node down. I opened a bug report here:

15

Yes, you are correct here. And I know about it too.

And I knew It from the beginning there is some kind of a bug.(But alas i wont be getting any credit for finding the issue) And you are correct it can occur in strange circumstances but you never know when this circumstances arise.

Can I contribute to resolve this bug and make pull request for it?

And a big thanks for helping out.

16

Apologies, I meant to link back to this thread but you beat me to it.

17

Ok .. not any issue .. just a beginner in elastic Search.. Will soon be releasing a PR for it.

1 Like
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.