Unassigned replica shards after creation of an index, even though there is enough space

Elastic Stack Elasticsearch
1

Hey,

I've been having a problem lately, first with ES 1.7X, and then with ES 5.4X, which didn't solve the problem.

Sometimes, when I create an index (more likely when I create a few indices in a row), I get unassigned replica shards, that never get assigned, not even after waiting a few hours.

If I do the following, it solves the problem and all replica shards get assigned, and index returns to green status -

  1. Update index settings to no replicas -
    PUT /INDEX_NAME/_settings
    {
    "number_of_replicas": 0
    }
  2. Wait for just a few seconds
  3. Update index settings back to 1 replica -
    PUT /INDEX_NAME/_settings
    {
    "number_of_replicas": 1
    }

These are the cluster details -

3 Master Nodes (1 Core, 3.5 RAM each)
3 Data Nodes (4 Core, 14 GB RAM each)

Storage - 100GB

Allocation and index settings -

For each index, number of shards - 3
For each index, number of replicas - 1
Total number of shards per node - 2

So with 3 data nodes and these settings, we should have a perfectly balanced cluster, where each node has exactly 2 shards of each index, and each index has a total of 6 shards (primary and replica nodes together), and that's why I don't understand why replica shards wouldn't get assigned.

Internet says most of the times it's because the replica shards have no where to be, because of a too low number of data nodes. But we have enough nodes and balancing rules to keep the whole cluster balanced, and it does get fixed once you update the settings yourself.

Any ideas why that would happen and how to solve it?

Thanks in advance for any help.

2

On 5.x I think that you should use the cluster allocation explain API to understand why the shards are unassigned. With that information, we can better debug what is happening here.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.