Fixing yellow health of cluster due to permission error

In my elastic search cluster, some nodes are red while some of them are yellow. The point is only beats indices are affected. So i executed curl localhost:9200/_cluster/allocation/explain

The output

{
	"index": "filebeat-7.6.0-2020.02.22",
	"shard": 0,
	"primary": false,
	"current_state": "unassigned",
	"unassigned_info": {
		"reason": "CLUSTER_RECOVERED",
		"at": "2020-02-29T23:19:30.698Z",
		"last_allocation_status": "no_attempt"
	},
	"can_allocate": "no",
	"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
	"node_allocation_decisions": [
		{
			"node_id": "THJOod5tSUGcF23uhLuKfw",
			"node_name": "node-1",
			"transport_address": "192.168.43.22:9300",
			"node_attributes": {
				"ml.machine_memory": "4136235008",
				"xpack.installed": "true",
				"ml.max_open_jobs": "20"
			},
			"node_decision": "no",
			"deciders": [
				{
					"decider": "same_shard",
					"decision": "NO",
					"explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[filebeat-7.6.0-2020.02.22][0], node[THJOod5tSUGcF23uhLuKfw], [P], s[STARTED], a[id=1jY1u__lRNKYGDibSFoWcg]]"
				}
			]
		}
	]
}

Screenshor of some indices

@tbhaxor The error from the cluster allocation explain API is pretty clear. The replica shard for index filebeat-7.6.0-2020.02.22 cannot be allocated on any nodes. It looks like your cluster is a one-node cluster and in this case the primary shard and its copy (replica shard) cannot be allocated on the same node.

This also explains why some of your indices are green because they do not have any replica shards configured.

Two options to move forward:

  • Add new nodes in your cluster so that the replica shards can be allocated. This would be the recommended approach.
  • Set the index.number_of_replicas to 0 for the respective indices. You can use the update index settings API. We generally recommend not to do this because you could be facing data loss if any failures happen with the nodes in your cluster.

I hope that helps.

Ok so in X/Y, the X is the number of shards and Y is the number of replica sets?

Yes @tbhaxor - that's right.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.