Unassigned replica shards with reason as INDEX_CREATED

[Aliasgar_Choolawala (Aliasgar)]

can anyone please help me ?
i have a 7 data node elasticsearch cluster(5 master nodes)

for an index, i have the following settings for routing:

"routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            },
            "total_shards_per_node" : "1"
          }
        },
        "refresh_interval" : "120s",
        "number_of_shards" : "7",
        "translog" : {
          "flush_threshold_size" : "1g",
          "retention" : {
            "size" : "1g"
          }
        },
"number_of_replicas" : "1",

all the replicas are UNASSIGNED with reason as "INDEX_CREATED" :

{
              "state" : "UNASSIGNED",
              "primary" : false,
              "node" : null,
              "relocating_node" : null,
              "shard" : 6,
              "index" : "pm2-logs-2020.06.19-000104",
              "recovery_source" : {
                "type" : "PEER"
              },
              "unassigned_info" : {
                "reason" : "INDEX_CREATED",
                "at" : "2020-06-19T14:01:38.699Z",
                "delayed" : false,
                "allocation_status" : "no_attempt"
              }
            }

[Vinayak_Sapre]

Hi Aliasgar,

Your index has 7 shards and 1 replica that makes 14 shards to allocate. Since total_shards_per_node is 1, you will need 14 data nodes, you have only 7 data nodes.

[Aliasgar_Choolawala (Aliasgar)]

thanks for your suggestion, @Vinayak_Sapre
i'll update the template settings

[Aliasgar_Choolawala (Aliasgar)]

hey @Vinayak_Sapre

after i made changes for total_shards_per_node, still out of 14 shards, one replica shard remains unassigned after index rollover.

this is not happening after every rollover, but happening frequently!

"6" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "xxxxxxxxxxxxxxxx",
              "relocating_node" : null,
              "shard" : 6,
              "index" : "pm2-logs-2020.06.23-000172",
              "allocation_id" : {
                "id" : "xxxxxxxxxxxxxxxx"
              }
            },
            {
              "state" : "UNASSIGNED",
              "primary" : false,
              "node" : null,
              "relocating_node" : null,
              "shard" : 6,
              "index" : "pm2-logs-2020.06.23-000172",
              "recovery_source" : {
                "type" : "PEER"
              },
              "unassigned_info" : {
                "reason" : "INDEX_CREATED",
                "at" : "2020-06-23T10:41:38.561Z",
                "delayed" : false,
                "allocation_status" : "no_attempt"
              }
            }
          ]

and due to this, the index status remains yellow, and is not green

[Aliasgar_Choolawala (Aliasgar)]

i managed to overcome this issue, by assigning "total_shards_per_node" as 3 instead of 2, since I noticed that while the rollover happens, the deletion and creation of replica shard(s) happens in parallel, and so at some point of time, one of the node(s) might have more than 2 shards for the index.

After changing the value to 3, now all the rollover indices are green.

I managed to get to this answer by this

 {
          "decider" : "awareness",
          "decision" : "NO",
          "explanation" : "there are too many copies of the shard allocated to nodes with attribute [aws_availability_zone], there are [2] total configured shard copies for this shard id and [3] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
        }

which is a part of the output of the query:

GET /_cluster/allocation/explain
{
  "index": "pm2-logs-2020.06.23-000186",
  "shard": 6,
  "primary": false
}