Unable to allocate .security index

[Jathin]

I have a alert on Kibana monitoring index stating a node got removed and got added later and I see the cluster in yellow state.
No running GET _cluster/allocation/explain i get .security index is not allocated as all nodes already have one.
here is the output of the call
same reason is mentioned for all nodes.

What actions need to be performed to get the cluster back to green?

{
  "index": ".security-6",
  "shard": 0,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "REPLICA_ADDED",
    "at": "2019-03-20T22:02:26.227Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "-KKqR4y_Rla0m_lfm2IwSw",
      "node_name": "[REDACTED]",
      "transport_address": "[REDACTED]",
      "node_attributes": {
        "ml.machine_memory": "64217174016",
        "ml.max_open_jobs": "20",
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security-6][0], node[-KKqR4y_Rla0m_lfm2IwSw], [R], s[STARTED], a[id=XateldKdTmOooDSzmFk_rQ]]"
        }
      ]

[Jathin]

Elasticsearch version is 6.2.3

[Jathin]

I have disabled allocation to 7 nodes using

put _cluster/settings
{
  "transient" : {
"cluster.routing.allocation.exclude._name" : "node names"
  }
}

And the a .security shard is not allocated to one of the nodes on that list. When i try to remove that node from the list and try to reroute failed shards it then tells me that it fails allocation one another shard and then the cycle continues.

[Jathin]

Got it fixed.

removed the allocation exclusion for nodes. and hence elasticsearch was able to figure out the allocation of the security index properly. and then it started to re balance the whole cluster.
I added back the exclusion and it took couple of hours to remove those shards from those nodes.
Those nodes are soon going to be terminated.

Thanks for reading