# Replicas allocation issue in a hot cold architecture
**URL:** https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953
**Category:** Elasticsearch
**Created:** [November 19, 2020, 9:23am UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953 "2020-11-19T09:23:34Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Travis](https://discuss.elastic.co/u/Travis)
#### Post date: [November 19, 2020, 9:23am UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/1 "2020-11-19T09:23:34Z")
Hello !
I have the following cluster :
- **node1** with attributes **box\_type: hot** and **zone: zone1**
- **node2** with attributes **box\_type: hot** and **zone: zone1**
- **node3** with attributes **box\_type: hot** and **zone: zone2**
- **node4** with attributes **box\_type: cold** and **zone: zone1**
- **node5** with attributes **box\_type: cold** and **zone: zone2**
Masters and Data : node1, node2, node 3 (all hot nodes)
Data only : node 4, node 5 (all cold nodes)
//
**On the three masters** , I set this configuration :
- cluster.routing.allocation.awareness.attributes: zone
- cluster.routing.allocation.awareness.force.zone.values: zone1,zone2
//
- Index template :
> ```
> {
> "filebeat-template-test" : {
> "order" : 0,
> "index_patterns" : [
> "filebeat-test-*"
> ],
> "settings" : {
> "index" : {
> "lifecycle" : {
> "name" : "mainpolicy",
> "rollover_alias" : "filebeat-test"
> },
> "routing" : {
> "allocation" : {
> "include" : {
> "box_type" : "hot"
> }
> }
> },
> "mapping" : {
> "total_fields" : {
> "limit" : "10000"
> }
> },
> [...]
>
> ```
ILM policy :
> {
> "mainpolicy" : {
> "version" : 3,
> "modified\_date" : "2020-11-06T08:00:47.215Z",
> "policy" : {
> "phases" : {
> "hot" : {
> "min\_age" : "0ms",
> "actions" : {
> "rollover" : {
> "max\_size" : "50gb",
> "max\_age" : "1d"
> },
> "set\_priority" : {
> "priority" : null
> }
> }
> },
> "delete" : {
> "min\_age" : "10d",
> "actions" : {
> "delete" : {
> "delete\_searchable\_snapshot" : true
> }
> }
> },
> "cold" : {
> "min\_age" : "7d",
> "actions" : {
> "allocate" : {
> "include" : { },
> "exclude" : { },
> "require" : {
> "box\_type" : "cold"
> }
> },
> "freeze" : { },
> "set\_priority" : {
> "priority" : null
Index settings :
> {
> "settings": {
> "index": {
> "lifecycle": {
> "name": "mainpolicy",
> "rollover\_alias": "filebeat-test",
> "indexing\_complete": "true"
> },
> "routing": {
> "allocation": {
> "include": {
> "box\_type": "hot"
> },
> "require": {
> "box\_type": "cold"
> }
> }
> },
> "mapping": {
> "total\_fields": {
> "limit": "10000"
> }
> },
> "refresh\_interval": "5s",
> "number\_of\_shards": "3",
> "provided\_name": "filebeat-test-000006",
> "max\_docvalue\_fields\_search": "200",
> [...]
After a restart of all nodes, I have the following error :
> {
> "index" : "filebeat-test-000006",
> "shard" : 1,
> "primary" : false,
> "current\_state" : "unassigned",
> "unassigned\_info" : {
> "reason" : "CLUSTER\_RECOVERED",
> "at" : "2020-11-19T08:06:18.506Z",
> "last\_allocation\_status" : "no\_attempt"
> },
> "can\_allocate" : "no",
> "allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
> "node\_allocation\_decisions" : [
> {
> "node\_id" : "2gjmhy4gT1-WKvvTqtn3Vg",
> "node\_name" : "node4",
> "transport\_address" : "192.168.1.11:9300",
> "node\_attributes" : {
> "xpack.installed" : "true",
> "box\_type" : "cold",
> "zone" : "zone1",
> "transform.node" : "true"
> },
> "node\_decision" : "no",
> "deciders" : [
> {
> "decider" : "filter",
> "decision" : "NO",
> "explanation" : """node does not match index setting [index.routing.allocation.include] filters [box\_type:"hot"]"""
> }
> ]
> },
> {
> "node\_id" : "3qvjjvnvTUajtoP0X4h97g",
> "node\_name" : "node1",
> "transport\_address" : "192.168.1.27:9300",
> "node\_attributes" : {
> "xpack.installed" : "true",
> "box\_type" : "hot",
> "zone" : "zone1",
> "transform.node" : "true"
> },
> "node\_decision" : "no",
> "deciders" : [
> {
> "decider" : "filter",
> "decision" : "NO",
> "explanation" : """node does not match index setting [index.routing.allocation.require] filters [box\_type:"cold"]"""
> }
> ]
> },
> {
> "node\_id" : "7Hp7eqbSQIqZSiH-s6rVdQ",
> "node\_name" : "node3",
> "transport\_address" : "192.168.1.29:9300",
> "node\_attributes" : {
> "xpack.installed" : "true",
> "box\_type" : "hot",
> "zone" : "zone2",
> "transform.node" : "true"
> },
> "node\_decision" : "no",
> "deciders" : [
> {
> "decider" : "filter",
> "decision" : "NO",
> "explanation" : """node does not match index setting [index.routing.allocation.require] filters [box\_type:"cold"]"""
> },
> {
> "decider" : "same\_shard",
> "decision" : "NO",
> "explanation" : "a copy of this shard is already allocated to this node [[filebeat-test-000006][1], node[7Hp7eqbSQIqZSiH-s6rVdQ], [P], s[STARTED], a[id=IjIUykz0Sm27cT-Wa4ZpfA]]"
> },
> {
> "decider" : "awareness",
> "decision" : "NO",
> "explanation" : "there are too many copies of the shard allocated to nodes with attribute [zone], there are [2] total configured shard copies for this shard id and [3] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
> }
> ]
> },
> {
> "node\_id" : "J0FDiT6mRLG7kyujZ1RC\_g",
> "node\_name" : "node2",
> "transport\_address" : "192.168.1.28:9300",
> "node\_attributes" : {
> "xpack.installed" : "true",
> "box\_type" : "hot",
> "zone" : "zone1",
> "transform.node" : "true"
> },
> "node\_decision" : "no",
> "deciders" : [
> {
> "decider" : "filter",
> "decision" : "NO",
> "explanation" : """node does not match index setting [index.routing.allocation.require] filters [box\_type:"cold"]"""
> }
> ]
> },
> {
> "node\_id" : "dgEgyoNmTSalZmqDx3mUoQ",
> "node\_name" : "node5",
> "transport\_address" : "192.168.1.10:9300",
> "node\_attributes" : {
> "xpack.installed" : "true",
> "box\_type" : "cold",
> "zone" : "zone2",
> "transform.node" : "true"
> },
> "node\_decision" : "no",
> "deciders" : [
> {
> "decider" : "filter",
> "decision" : "NO",
> "explanation" : """node does not match index setting [index.routing.allocation.include] filters [box\_type:"hot"]"""
> },
> {
> "decider" : "awareness",
> "decision" : "NO",
> "explanation" : "there are too many copies of the shard allocated to nodes with attribute [zone], there are [2] total configured shard copies for this shard id and [3] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
> }
> ]
> }
> ]
> }
**It looks like only replicas shards which have to be in cold phase are concerned.**
Example of a concerned index :
- Shards allocation for a index without issue (in hot phase) :
zone1 : replica 1 // replica 2 // primary 0
zone2 : replica 0 // primary 1 // primary 2
- Shards allocation for a index with issue (have to be in cold phase) :
zone1: primary 0
zone2: primary 1 // primary 2
Thanks for your help !
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [November 19, 2020, 12:18pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/2 "2020-11-19T12:18:32Z")
> [@Travis](#):
>
> ```auto
> "explanation" : """node does not match index setting [index.routing.allocation.include] filters [box_type:"hot"]"""
> ...
> "explanation" : """node does not match index setting [index.routing.allocation.require] filters [box_type:"cold"]"""
>
> ```
You have conflicting settings for this index, requiring it to be on both a `hot` and a `cold` node.
Additionally, this message indicates that Elasticsearch sees three zones, not two:
> [@Travis](#):
>
> `attribute [zone], there are ... [3] total attribute values`
---
### Author:  [@Travis](https://discuss.elastic.co/u/Travis)
#### Post date: [November 19, 2020, 12:45pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/3 "2020-11-19T12:45:02Z")
> [@DavidTurner](#):
>
> You have conflicting settings for this index, requiring it to be on both a `hot` and a `cold` node.
How is it posible ?
> [@DavidTurner](#):
>
> Additionally, this message indicates that Elasticsearch sees three zones, not two:
Same question. As I verified my settings and there is only two zones
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [November 19, 2020, 1:15pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/4 "2020-11-19T13:15:48Z")
> [@Travis](#):
>
> How is it posible ?
Not sure how to answer that apart from quoting the same messages again:
```auto
"explanation" : """node does not match index setting [index.routing.allocation.include] filters [box_type:"hot"]"""
...
"explanation" : """node does not match index setting [index.routing.allocation.require] filters [box_type:"cold"]"""
```
i.e. this index has both `index.routing.allocation.include.box_type: hot` and `index.routing.allocation.require.box_type: cold`, which don't make sense together.
> [@Travis](#):
>
> Same question. As I verified my settings and there is only two zones
Don't know, sorry, but Elasticsearch definitely sees 3.
---
### Author:  [@Travis](https://discuss.elastic.co/u/Travis)
#### Post date: [November 19, 2020, 5:10pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/5 "2020-11-19T17:10:30Z")
> [@DavidTurner](#):
>
> i.e. this index has both `index.routing.allocation.include.box_type: hot` and `index.routing.allocation.require.box_type: cold` , which don't make sense together.
You mean that in a hot/cold architecture, I shouldn't see this in index settings :
> [@Travis](#):
>
> "routing": {
> "allocation": {
> "include": {
> "box\_type": "hot"
> },
> "require": {
> "box\_type": "cold"
> }
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [November 19, 2020, 6:37pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/6 "2020-11-19T18:37:04Z")
That's correct (unless of course you have any nodes that are both hot and cold)
---
### Author:  [@Travis](https://discuss.elastic.co/u/Travis)
#### Post date: [November 20, 2020, 2:45pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/7 "2020-11-20T14:45:29Z")
@DavidTurner
As you said, I have a conflict and I think this is between ILM set in Kibana and my index template :
In Kibana, you can select a node attribute for the cold phase, this is what I did. But, for the hot phase, you can't. And that's why I set this setting in my index template. But this apply to _all_ shards.
So, on one side, I ask Elasticsearch to route _all shards_ to nodes with box attributes hot, and on the other side, I ask Elasticsearch (using ILM policy created in Kibana) that shards with a min\_age of 7 days have to be in cold phase and so cold nodes.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 18, 2020, 2:45pm UTC](https://discuss.elastic.co/t/replicas-allocation-issue-in-a-hot-cold-architecture/255953/8 "2020-12-18T14:45:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.