# Allocation awarenes vs data tiering problems

**URL:** <https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820>\
**Category:** Elasticsearch\
**Created:** [June 3, 2025, 12:46pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820 "2025-06-03T12:46:01Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 3, 2025, 12:46pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/1 "2025-06-03T12:46:01Z")

</div>

Minimal setup is like this:

| Node | Data tier | Rack |
| --- | --- | --- |
| 1 | data\_hot,data\_content | A |
| 2 | data\_hot,data\_content | B |
| 3 | data\_hot,data\_content | B |
| 4 | data\_cold | A |
| 5 | data\_cold | B |

- Node 1 failed.
- I expected elastic create new replicas in other rack B node.
- Awareness decider refused: `there are [2] copies of this shard and [2] values for attribute [rack] ([A, B] from nodes in the cluster and no forced awareness) so there may be at most [1] copies of this shard allocated to nodes with each value, but (including this copy) there would be [2] copies allocated to nodes with [node.attr.rack: B]`

Is my configuration wrong or allocation awareness works badly with data tiering?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 3, 2025, 12:59pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/2 "2025-06-03T12:59:32Z")

</div>

> [@nisow95612](#):
>
> Is my configuration wrong or allocation awareness works badly with data tiering?

What does your configuration looks like? You didn't share anything from your `elasticsearch.yml`.

Are you using [forced awareness](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/shard-allocation-awareness.html#forced-awareness) on the `rack` attribute, right?

If so, this is working as designed.

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 3, 2025, 1:18pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/3 "2025-06-03T13:18:20Z")

</div>

> [@leandrojmp](#):
>
> Are you using [forced awareness](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/shard-allocation-awareness.html#forced-awareness) on the `rack` attribute, right?

No force awareness just like awareness decider says.  
Can awareness decider maybe count rack A alive because of cold node 4?

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 3, 2025, 1:25pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/4 "2025-06-03T13:25:14Z")

</div>

> [@leandrojmp](#):
>
> What does your configuration looks like? You didn't share anything from your `elasticsearch.yml`

| node.name | node.roles | node.attr.rack |
| --- | --- | --- |
| n1 | data\_hot,data\_content,master | A |
| n2 | data\_hot,data\_content,master | B |
| n3 | data\_hot,data\_content,master | B |
| n4 | data\_cold | A |
| n5 | data\_cold | B |

`cluster.routing.allocation.awareness.attributes: rack`

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 3, 2025, 1:31pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/5 "2025-06-03T13:31:30Z")

</div>

You need to provide more context.

How many primary and replicas does this index have? Can you share the entire allocation explain? And what is the tier preference for this index? Does the index have any unassigned shard?

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 3, 2025, 1:59pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/6 "2025-06-03T13:59:24Z")

</div>

OK. And btw thank you for trying to help!

Index is 1 shard, 1 replica, data\_hot tier.

> **This in create index settings**
>
> ```auto
> "index.routing.allocation.include._tier_preference": "data_hot",
> "index.number_of_shards": "1",
> "index.number_of_replicas": "1"
> 
> ```

Before `n1` fail was one copy on `n1` and other copy on `n2`.  
After `n1` fail I have one copy on `n2` and other copy not allocated.

> **Full output for n3**
>
> ```auto
> {
> "node_id" : " *****",
> "node_name" : "n3",
> "transport_address" : " *****",
> "node_attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "deciders" : [
> {
> "decider" : "awareness",
> "decision" : "NO",
> "explanation" : "there are [2] copies of this shard and [2] values for attribute [rack] ([A, B] from nodes in the cluster and no forced awareness) so there may be at most [1] copies of this shard allocated to nodes with each value, but (including this copy) there would be [2] copies allocated to nodes with [node.attr.rack: B]"
> }
> ]
> },
> 
> ```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 3, 2025, 2:09pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/7 "2025-06-03T14:09:17Z")

</div>

Please share the full responses not just part of it, it is really complicated to troubleshoot without having the full context.

Also share the result of running `GET _cat/shards/name-of-the-index?v` on Kibana Dev Tools.

From what you shared it is not sure if there is any issue or if it is working as designed.

Also, which version are you running? You didn´t say.

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 4, 2025, 8:04am UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/8 "2025-06-04T08:04:25Z")

</div>

OK this is dumb. Explain allocation api explains different when I send shard name or not.

> **Explain allocation without shard name**
>
> ```auto
> {
> "index" : "newdatab_80",
> "shard" : 0,
> "primary" : false,
> "current_state" : "unassigned",
> "unassigned_info" : {
> "reason" : "NODE_LEFT",
> "at" : "2025-06-02T15:39:21.158Z",
> "details" : "node_left [*****]",
> "last_allocation_status" : "no_attempt"
> },
> "can_allocate" : "no",
> "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
> "node_allocation_decisions" : [
> {
> "node_id" : " *****",
> "node_name" : "n4",
> "transport_address" : "192.168.0.4:9300",
> "node_attributes" : {
> "rack" : "A",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "deciders" : [
> {
> "decider" : "data_tier",
> "decision" : "NO",
> "explanation" : "index has a preference for tiers [data_hot] and node does not meet the required [data_hot] tier"
> }
> ]
> },
> {
> "node_id" : " *****",
> "node_name" : "n3",
> "transport_address" : "192.168.0.3:9300",
> "node_attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "deciders" : [
> {
> "decider" : "awareness",
> "decision" : "NO",
> "explanation" : "there are [2] copies of this shard and [2] values for attribute [rack] ([A, B] from nodes in the cluster and no forced awareness) so there may be at most [1] copies of this shard allocated to nodes with each value, but (including this copy) there would be [2] copies allocated to nodes with [node.attr.rack: B]"
> }
> ]
> },
> {
> "node_id" : " *****",
> "node_name" : "n2",
> "transport_address" : "192.168.0.2:9300",
> "node_attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "deciders" : [
> {
> "decider" : "same_shard",
> "decision" : "NO",
> "explanation" : "a copy of this shard is already allocated to this node [[newdatab_80][0], node[*****], [P], s[STARTED], a[id=yHKkZBPcm7zHY5HZdnBCTh]]"
> },
> {
> "decider" : "awareness",
> "decision" : "NO",
> "explanation" : "there are [2] copies of this shard and [2] values for attribute [rack] ([A, B] from nodes in the cluster and no forced awareness) so there may be at most [1] copies of this shard allocated to nodes with each value, but (including this copy) there would be [2] copies allocated to nodes with [node.attr.rack: B]"
> }
> ]
> },
> {
> "node_id" : " *****",
> "node_name" : "n5",
> "transport_address" : "192.168.0.5:9300",
> "node_attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "deciders" : [
> {
> "decider" : "data_tier",
> "decision" : "NO",
> "explanation" : "index has a preference for tiers [data_hot] and node does not meet the required [data_hot] tier"
> }
> ]
> }
> ]
> }
> 
> ```

> **Explain allocation with shard name**
>
> ```auto
> {
> "index" : "newdatab_80",
> "shard" : 0,
> "primary" : true,
> "current_state" : "started",
> "current_node" : {
> "id" : " *****",
> "name" : "n2",
> "transport_address" : "192.168.0.2:9300",
> "attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "weight_ranking" : 4	
> },
> "can_remain_on_current_node" : "yes",
> "can_rebalance_cluster" : "no",
> "can_rebalance_cluster_decisions" : [
> {
> "decider" : "rebalance_only_when_active",
> "decision" : "NO",
> "explanation" : "rebalancing is not allowed until all replicas in the cluster are active"
> },
> {
> "decider" : "cluster_rebalance",
> "decision" : "NO",
> "explanation" : "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
> }
> ],
> "can_rebalance_to_other_node" : "no",
> "rebalance_explanation" : "rebalancing is not allowed, even though there is at least one node on which the shard can be allocated",
> "node_allocation_decisions" : [
> {
> "node_id" : " *****",
> "node_name" : "n3",
> "transport_address" : "192.168.0.3:9300",
> "node_attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "yes",
> "weight_ranking" : 3
> },
> {
> "node_id" : " *****",
> "node_name" : "n4",
> "transport_address" : "192.168.0.4:9300",
> "node_attributes" : {
> "rack" : "A",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "weight_ranking" : 1,
> "deciders" : [
> {
> "decider" : "data_tier",
> "decision" : "NO",
> "explanation" : "index has a preference for tiers [data_hot] and node does not meet the required [data_hot] tier"
> }
> ]
> },
> {
> "node_id" : " *****",
> "node_name" : "n5",
> "transport_address" : "192.168.0.5:9300",
> "node_attributes" : {
> "rack" : "B",
> "xpack.installed" : "true",
> "transform.node" : "false"
> },
> "node_decision" : "no",
> "weight_ranking" : 2,
> "deciders" : [
> {
> "decider" : "data_tier",
> "decision" : "NO",
> "explanation" : "index has a preference for tiers [data_hot] and node does not meet the required [data_hot] tier"
> }
> ]
> }
> ]
> }
> 
> ```

> **Explain shards**
>
> ```auto
> index shard prirep state docs store ip node
> newdatab_80 0 p STARTED 1283 139.1kb 192.168.0.2 n2
> newdatab_80 0 r UNASSIGNED
> 
> ```

Version 7.17.28. because one of my apps still refuses ES8 ☹

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 4, 2025, 12:43pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/9 "2025-06-04T12:43:25Z")

</div>

> [@nisow95612](#):
>
> Explain allocation api explains different when I send shard name or not

It is because the first explain will get a random shard and explain the allocation for it, normally it gets a random unassigned shard, which is the case, this first explain is for the replica shard.

The second one is for the primary shard.

I do not use awereness, but I'm not sure what is the issue here, from the documentation it seems that it should allocate the shard on the other node in rack B, this is mentioned in the forced awareness part.

> With this example configuration, if you have two nodes with `node.attr.zone` set to `zone1` and an index with `number_of_replicas` set to `1` , Elasticsearch allocates all the primary shards but none of the replicas. It will assign the replica shards once nodes with a different value for `node.attr.zone` join the cluster. **In contrast, if you do not configure forced awareness, Elasticsearch will allocate all primaries and replicas to the two nodes even though they are in the same zone.**

So, the primary is in node `n2`, so with just the awereness configured without using forced awereness I would expect the replica to be allocated to the node `n3`, not sure why it is not doing that.

You will need to see if someone from elastic can provide more context or maybe open an issue on github.

From the documentation I would expect the replica to be allocated to node 3, but maybe the documentation is wrong.

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 4, 2025, 1:18pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/10 "2025-06-04T13:18:29Z")

</div>

Yeah. So thank you for confirming it should allocate on `n3` like this.

> **Off topic**
>
> Microsoft account is unfortunately big no for me so I have to hope somebody from elastic comes?

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 4, 2025, 2:51pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/12 "2025-06-04T14:51:39Z")

</div>

> **Off topic**
>
> Update: Here is real message I send: Microsoft owns website (github) that leandrojmp recommended for contacting support. Microsoft requires their account to log in.
> 
> Means: It is not because they own github. It is because they require microsoft accounts which are pain.

---

<div class="post-metadata">

**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\
**Post date:** [June 4, 2025, 3:13pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/13 "2025-06-04T15:13:06Z")

</div>

Not getting into argument here, in future please answer on forum, not via PM please..

Your position is clear. You might have hit a bug, but you’re not prepared to report it on GitHub for reasons given. This is certainly your right.

I’ve helped resolve a bunch of problems on the forum. Others have done far more than me. It’s rare the people who raise queries are as inflexible.

You could also open a support ticket if you are paying customer. But, wild guess, you’re not?

I hope someone does answer your query cos it’s an interesting possible-bug.

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 4, 2025, 3:22pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/14 "2025-06-04T15:22:41Z")

</div>

> **Off topic**
>
> OK, wil remember for next time. Did not want off topic spam in conversation.  
> Sorry for being unable to report it on GitHub. But lot of iflexibility is with microsoft that makes their accounts pain. I guess you are big tech customer so they go easier on you?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 4, 2025, 3:53pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/15 "2025-06-04T15:53:18Z")

</div>

Not sure if I understand, but you can create a Github account using any email account you have, you do not need a Microsoft Account to have an account on Github.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 18, 2025, 1:37pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/17 "2025-06-18T13:37:16Z")

</div>

> [@nisow95612](#):
>
> Microsoft account is unfortunately big no for me so I have to hope somebody from elastic comes?

Yeah that's fine - it's of course helpful if you can report issues on Github but not everyone is willing or able to do that and it's not at all productive to debate this point.

In this case I think what you're describing is fundamentally the same as this issue:

> <https://github.com/elastic/elasticsearch/issues/52374>
>
> Today if you are using allocation awareness and you wish to decommission an enti…re zone, you may not be able to gracefully vacate all shards from this zone since allocation awareness computes the maximum number of shards per zone without regard for any allocation filters, and these constraints may conflict.
> 
> We will address this with #49064, but we may like to consider having allocation awareness respect certain cluster-wide allocation exclusions too, as a special case to support zone-wide vacation. We saw one case where this might have been useful recently, so I'm opening this to collect other evidence that this would be useful.
> 
> Relates #8178, #49064.

---

<div class="post-metadata">

**Author:** ![nisow95612](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@nisow95612](https://discuss.elastic.co/u/nisow95612)\
**Post date:** [June 18, 2025, 2:54pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/18 "2025-06-18T14:54:45Z")

</div>

Yes, looks right. Cause is in how elastic counts available zones.  
Thank you for stopping by!

Making my lab cluster hit this limitation was surprisingly hard so I don't think it is big limitation. By the I also noticed awareness can cause "grid lock" that stops rebalancing but I don't think it is worth another topic.

> **"grid lock" in awareness attributes**
>
> | Node | attr.rack | attr.power |
> | --- | --- | --- |
> | n1 | r1 | pA |
> | n2 | r1 | pB |
> | n3 | r2 | pA |
> | n4 | r2 | pB |
> 
> Other view:
> 
> | power/rack | r1 | r2 |
> | --- | --- | --- |
> | pA | n1 | n3 |
> | pB | n2 | n4 |
> 
> A index with one shard and one replica will be allowed on n1+n4 or n2+n3 but will not be able to rebalance to other pair of nodes because intermediate steps violate awareness.
> 
> I hoped to create helper "migrator" zone with redundant power but found no way to do it.  
> Do you know how to make node member of both zones -\> good if either power A or B?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 18, 2025, 3:03pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/19 "2025-06-18T15:03:02Z")

</div>

Folks lets keep on track...  
Looks like an understanding / solution has been provided to an interesting issue.  
I am going to ignore most of the rest of the flagged posts... I hide some...  
In the future if a user does not want to create a github issue that is OK.  
At the same time commenting on behavior / opinions etc. should be avoided.  
PM unless specifically requested should also be avoided.  
Thanks everyone appreciate the interaction, lets keep it technical.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 18, 2025, 3:03pm UTC](https://discuss.elastic.co/t/allocation-awarenes-vs-data-tiering-problems/378820/20 "2025-06-18T15:03:15Z")

</div>