# New replica are not getting assigned **URL:** \ **Category:** Elasticsearch\ **Created:** [July 17, 2017, 6:44pm UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450 "2017-07-17T18:44:34Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 17, 2017, 6:44pm UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/1 "2017-07-17T18:44:34Z")
I use kopf for visualization, and I tried changing the number of replica setting for an index from 17 to 20, making the replication group of 21 using kopf. (total 20 primary shards + 3 availability zones) The observation is, it assigns some number of replica's (mainly group of 60) but others stay unassigned. Any pointers on debugging this issue ? Elasticsearch version 1.7 ---
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [July 17, 2017, 8:06pm UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/2 "2017-07-17T20:06:45Z")
You can try [Explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html), which can tell you why it could not assign some shards. ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 18, 2017, 9:45pm UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/3 "2017-07-18T21:45:04Z")
I don't think this API is available in 1.7. Any other way to figureout the issue ? ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 18, 2017, 9:51pm UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/4 "2017-07-18T21:51:32Z")
To share more information, it mainly fails on "too many shards on nodes for attribute: [aws\_availability\_zone]" but I can clearly see there are many hosts in availability zone without the shards. The hack that I use to force allocate is, I increase the number of shards to relatively high value and cluster picks bunch of shards for allocation, I then reduce the replicas back to what is required and this solves the problem of unassigned shards. The process is very annoying when you have many indices 😕 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 18, 2017, 11:01pm UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/5 "2017-07-18T23:01:22Z")
> [@Dhara\_Desai](#): > > I tried changing the number of replica setting for an index from 17 to 20 Why do you have so many replicas? ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 1:48am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/6 "2017-07-19T01:48:36Z")
becasue we need more replicas for a business requirement, is there an issue with more number of replicas in 1.7 ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 19, 2017, 1:49am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/7 "2017-07-19T01:49:32Z")
No, it's just unusual to see many people running that many 🙂 ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 6:52am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/8 "2017-07-19T06:52:17Z")
We have following settings for force awareness cluster.routing.allocation.awareness.force.availability\_zone.values: zone1, zone2, zone3 cluster.routing.allocation.awareness.attributes: availability\_zone This will get applied to assigning and relocating shards, but will it include new replicas added to the replication group while calculating "shardPerAttribute"? ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 7:37am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/9 "2017-07-19T07:37:23Z")
adding a little more analysis, with 11 replica + 20 primary = total 240 shards Zone 1 - 73 assigned / 7 unassigned Zone 2 - 65 assigned / 15 unassigned Zone 3 - 78 assigned / 2 unassigned I tried manually rerouting one of the shards to host in each zone NO(too many shards on nodes for attribute: [availability\_zone] I also ran reroute with explain to get unassigned\_info, "reason": "NODE\_LEFT" which is expected. I am curious, what happens with the number of hosts in all these zones are not equal? will that create any imbalance in assigning shards ? our index setting for "total\_shards\_per\_node" is default ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 19, 2017, 7:40am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/10 "2017-07-19T07:40:57Z")
I am a bit confused. How many indices do these 240 shards belong to? How many primary shards do each index have? What is the number of replicas set to for these indices? How many data nodes do you have per availability zone? ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 7:45am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/11 "2017-07-19T07:45:18Z")
This is the analysis of 1 index with 20 primary and 11 replica and around 80 data nodes in each zone. we do have more indices other than the mentioned. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 19, 2017, 7:56am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/12 "2017-07-19T07:56:24Z")
If I have understood your configuration correctly, I would expect Elasticsearch to only assign the primary shard and 2 replicas of that shard as you have defined awareness of 3 zones. That is the purpose of shard allocation awareness according to the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-cluster.html#allocation-awareness). Do all nodes have the allocation awareness parameters configured? ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 8:04am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/13 "2017-07-19T08:04:50Z")
Yes all nodes have allocation awareness parameter set, I verified that. Are you saying with 3 zones, we can only have 1primary + 2 replica setting? what if we have more replicas, what is the expected behavior ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 19, 2017, 8:10am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/14 "2017-07-19T08:10:15Z")
I would expect those replicas to be unassigned. If you wanted to have 5 replicas (6 copies of each shard), you could divide up a zone into parts, e.g. `zone1a`, `zone1b`, `zone2a`, `zone2b`, `zone3a` and `zone3b`. If you leave out or alter the forced allocation parameter, Elasticsearch will try to allocate one shard per zone and will now be able to place one primary shard and 5 replicas. This is quite well explained in the [example given here](https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-cluster.html#forced-awareness). ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 8:13am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/15 "2017-07-19T08:13:31Z")
That makes sense, Ill have to revist our Elasticsearch awareness settings. ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 8:35am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/16 "2017-07-19T08:35:58Z")
I am curious, with current force settings if I look at replication group of shard 3, it shoud have 9 unassigned shards but all are assigned evenly accross zones. Which is strange and not an expected behavior ! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 19, 2017, 8:42am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/17 "2017-07-19T08:42:30Z")
That is what surprised me too, and why I asked if all nodes have all parameters correctly set. It does sound strange. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 19, 2017, 8:45am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/18 "2017-07-19T08:45:10Z")
> [@Dhara\_Desai](#): > > "too many shards on nodes for attribute: [aws\_availability\_zone]" Am curious where this error comes from, given that you have specified the allocation awareness attribute as just `availability_zone`. Is there a mismatch in the configuration? ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 8:46am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/19 "2017-07-19T08:46:06Z")
As per this conversation, I understood that removing "cluster.routing.allocation.awareness.force.availability\_zone.values: zone1, zone2, zone3" might resolve this issue. Ill quickly test it as I dont see the need of this setting for now, because we need more than 2 replicas for sure. ---
**Author:** ![Dhara\_Desai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhara_desai/32/15760_2.png) [@Dhara\_Desai](https://discuss.elastic.co/u/Dhara_Desai)\ **Post date:** [July 19, 2017, 8:47am UTC](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450/20 "2017-07-19T08:47:46Z")
> [@Christian\_Dahlqvist](#): > > Am curious where this error comes from, given that you have specified the allocation awareness attribute as just availability\_zone. Is there a mismatch in the configuration? Sorry about that, there is no missmatch. The attribute is aws\_availability\_zone. And I observed this error when I run reroute for a perticular shard [Next page](https://discuss.elastic.co/t/new-replica-are-not-getting-assigned/93450.md?page=2)