# Change topic from :Bad logic for allocation shard due to data tier arch -\> case with node roles syntax in elastic \>8.0.0

**URL:** <https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [March 21, 2022, 8:12am UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168 "2022-03-21T08:12:06Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 21, 2022, 8:12am UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/1 "2022-03-21T08:12:06Z")

</div>

Hi  
Today I observed not expected logic for allocation shards.  
Under configuration I've data tier structure like  
coordinator nodes - are responsible for taking data traffic  
es\_data\_ssd\* nodes - are storing hot phase (ILP was also configured and data node has such hot role)  
es\_data\_hdd\* nodes - are taking part for store warm phase  
Why Elasticsearch are using all node for allaction data in one time for all nodes even on coordinator?

Elasticsearch build on **ver 7.16.1**

```auto
logstash-data_test_load-2022.03.21 9 p STARTED 39956 28945 10.0.9.67 es_data_hdd_2_2
logstash-data_test_load-2022.03.21 10 p STARTED 39838 28850 10.0.9.89 es_data_hdd_5_2
logstash-data_test_load-2022.03.21 7 p STARTED 39924 28799 10.0.9.44 es_data_ssd_1_1
logstash-data_test_load-2022.03.21 11 p STARTED 40027 28948 10.0.9.118 es_data_ssd_1_3
logstash-data_test_load-2022.03.21 5 p STARTED 39717 28775 10.0.9.48 es_data_ssd_5_1
logstash-data_test_load-2022.03.21 12 p STARTED 40039 28978 10.0.9.120 es_data_hdd_4_3
logstash-data_test_load-2022.03.21 1 p STARTED 39897 28970 10.0.9.124 es_data_ssd_5_3
logstash-data_test_load-2022.03.21 13 p STARTED 40517 29312 10.0.9.97 es_data_hdd_6_2
logstash-data_test_load-2022.03.21 2 p STARTED 39974 28913 10.0.9.79 es_data_ssd_5_2
logstash-data_test_load-2022.03.21 14 p STARTED 40114 29002 10.0.9.85 es_data_ssd_1_2
logstash-data_test_load-2022.03.21 8 p STARTED 40102 29040 10.0.9.130 es_data_hdd_6_3
logstash-data_test_load-2022.03.21 6 p STARTED 39736 28825 10.0.9.56 es_data_hdd_2_1
logstash-data_test_load-2022.03.21 3 p STARTED 39874 28838 10.0.9.71 es_data_hdd_4_2
logstash-data_test_load-2022.03.21 4 p STARTED 40178 29054 10.0.9.93 es_coordination_2
logstash-data_test_load-2022.03.21 0 p STARTED 39873 28892 10.0.9.25 es_coordination_1

```

```auto
name node.role
es_coordination_1 cdfhimrstw
es_coordination_2 cdfhimrstw
es_coordination_3 cdfhimrstw
es_data_hdd_1_1 sw
es_data_hdd_1_2 sw
es_data_hdd_1_3 sw
es_data_hdd_2_1 sw
es_data_hdd_2_2 sw
es_data_hdd_2_3 sw
es_data_hdd_3_1 sw
es_data_hdd_3_2 sw
es_data_hdd_3_3 sw
es_data_hdd_4_1 sw
es_data_hdd_4_2 sw
es_data_hdd_4_3 sw
es_data_hdd_5_1 sw
es_data_hdd_5_2 sw
es_data_hdd_5_3 sw
es_data_hdd_6_1 sw
es_data_hdd_6_2 sw
es_data_hdd_6_3 sw
es_data_hdd_7_1 c
es_data_hdd_7_2 c
es_data_hdd_7_3 c
es_data_hdd_8_1 c
es_data_hdd_8_2 c
es_data_hdd_8_3 c
es_data_hdd_9_1 c
es_data_hdd_9_2 c
es_data_hdd_9_3 c
es_data_ssd_1_1 hs
es_data_ssd_1_2 hs
es_data_ssd_1_3 hs
es_data_ssd_2_1 hs
es_data_ssd_2_2 hs
es_data_ssd_2_3 hs
es_data_ssd_3_1 hs
es_data_ssd_3_1_ingest i
es_data_ssd_3_2 hs
es_data_ssd_3_2_ingest i
es_data_ssd_3_3 hs
es_data_ssd_3_3_ingest i
es_data_ssd_4_1 hs
es_data_ssd_4_2 hs
es_data_ssd_4_3 hs
es_data_ssd_5_1 hs
es_data_ssd_5_2 hs
es_data_ssd_5_3 hs
es_master_1_1 m
es_master_1_2 m
es_master_1_3 m
es_master_2_1 m
es_master_2_2 m
es_master_2_3 m

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 21, 2022, 8:35am UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/2 "2022-03-21T08:35:40Z")

</div>

> [@INS](#):
>
> `es_coordination_1 cdfhimrstw`

Your coordination nodes have all roles assigned and therefore hold data.

> [@INS](#):
>
> `es_data_hdd_1_1 sw`

The warm data nodes are designated as warm as well as content nodes. Wonder if the contant node status is having an undesired impact here? I would recommend them to just warm nodes.

> [@INS](#):
>
> `es_data_ssd_1_1 hs`

Yout hot data nodes are also content nodes. I would recommend making them pure hot data nodes and see if that has any impact.

> [@INS](#):
>
> ```auto
> es_master_1_1 m
> es_master_1_2 m
> es_master_1_3 m
> es_master_2_1 m
> es_master_2_2 m
> es_master_2_3 m
> 
> ```

What is the rationale behind having 6 dedicated master nodes? Usually 3 is recommended and sufficient.

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 21, 2022, 9:17am UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/3 "2022-03-21T09:17:41Z")

</div>

Thanks for checking tips. So my cluster consists with 3 hosts. On each host I have 18 nodes as You can see in my first comment. Therefore my intention was to have a redundancy cluster for master nodes. Do You have any suggest? this point? It could be one master node per host?  
So at least how I should define

`- node.roles=`  
for cooridnator node?

> **[Node | Elasticsearch Guide \[8.1\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-only-node)**

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 21, 2022, 8:59pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/4 "2022-03-21T20:59:05Z")

</div>

Can someone from Elastic Team Member explain how it should be configure coordinator node from roles perspective (it shouldn't store any data)

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 21, 2022, 10:35pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/5 "2022-03-21T22:35:01Z")

</div>

From the documentation [link](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/modules-node.html#coordinating-only-node) that you shared.

> To create a dedicated coordinating node, set:

```auto
node.roles: []

```

So, you need to set the `node.roles` as an empty array and your node will be a coordinating node only.

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 21, 2022, 10:58pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/6 "2022-03-21T22:58:47Z")

</div>

ha but it was defined as in documentation and it's store the data? why?

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 22, 2022, 9:14am UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/7 "2022-03-22T09:14:20Z")

</div>

coordinator node should be responsible for load balance traffic to others nodes (but not store data)? I'm right?

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 22, 2022, 12:44pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/8 "2022-03-22T12:44:24Z")

</div>

" **If you take away the ability to be able to handle master duties, to hold data, and pre-process documents, then you are left with a _coordinating_ node** that can only route requests, handle the search reduce phase, and distribute bulk indexing. Essentially, coordinating only nodes behave as smart load balancers.

Coordinating only nodes can benefit large clusters by offloading the coordinating node role from data and master-eligible nodes. They join the cluster and receive the full [cluster state](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-state.html), like every other node, and they use the cluster state to route requests directly to the appropriate place(s)"

Can elastic fix this documentation. It doesn't make sens. I've double check and it brings hold data.  
with "- node.roles=

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 22, 2022, 1:29pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/9 "2022-03-22T13:29:06Z")

</div>

Please share your entire `elasticsearch.yml` for your coordinating nodes.

It needs to be set exactly as:

```auto
node.roles: []

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 22, 2022, 2:03pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/10 "2022-03-22T14:03:10Z")

</div>

so it was deploy from yaml file

```auto
version: "3.7"

services:
  es_coordination_1:
    container_name: es_coordination_1
    image: priv_repo/elk-docker/elasticsearch/elasticsearch:8.1.0
    user: elasticsearch
    environment:
      - node.name=es_coordination_1
      - xpack.ml.enabled=false
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - xpack.security.http.ssl.key=certs/es_coordination_1/es_coordination_1.key
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.certificate=certs/es_coordination_1/es_coordination_1.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/es_coordination_1/es_coordination_1.crt
      - xpack.security.transport.ssl.key=certs/es_coordination_1/es_coordination_1.key
      - xpack.license.self_generated.type=basic
      - ELASTIC_USERNAME=elastic
      - ELASTIC_PASSWORD=changeme
      - cluster.name=elk_cluster
      - discovery.seed_hosts=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_1_3,es_master_2_3,es_data_ssd_3_1_ingest,es_data_ssd_1_1,es_data_ssd_2_1,es_data_ssd_3_1,es_data_ssd_4_1,es_data_ssd_5_1,es_data_hdd_1_1,es_data_hdd_2_1,es_data_hdd_3_1,es_data_hdd_4_1,es_data_hdd_5_1,es_data_hdd_6_1,es_data_hdd_7_1,es_data_hdd_8_1,es_data_hdd_9_1,es_coordination_2,es_master_1_2,es_master_2_2,es_data_ssd_3_2_ingest,es_data_ssd_1_2,es_data_ssd_2_2,es_data_ssd_3_2,es_data_ssd_4_2,es_data_ssd_5_2,es_data_hdd_1_2,es_data_hdd_2_2,es_data_hdd_3_2,es_data_hdd_4_2,es_data_hdd_5_2,es_data_hdd_6_2,es_data_hdd_7_2,es_data_hdd_8_2,es_data_hdd_9_2,es_coordination_3,es_master_1_3,es_master_2_3,es_data_ssd_3_3_ingest,es_data_ssd_1_3,es_data_ssd_2_3,es_data_ssd_3_3,es_data_ssd_4_3,es_data_ssd_5_3,es_data_hdd_1_3,es_data_hdd_2_3,es_data_hdd_3_3,es_data_hdd_4_3,es_data_hdd_5_3,es_data_hdd_6_3,es_data_hdd_7_3,es_data_hdd_8_3,es_data_hdd_9_3
      - cluster.initial_master_nodes=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_1_3,es_master_2_3
      - ingest.geoip.downloader.enabled=false
      - bootstrap.memory_lock=true
      - node.roles= 

```

however this syntax brings not expected configuration for node roles so it must be some workaround for that

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 22, 2022, 2:05pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/11 "2022-03-22T14:05:44Z")

</div>

> <https://github.com/elastic/go-ucfg/issues/188>
>
> When parsing empty arrays from YAML to config objects and unpacking into an unty…ped representation like \`map\[string\]interface{}\` arrays in the original source are turned into \`nil\`. Marshalling back to YAML this turns into \`null\` and is thereby not round-trippeable through ucfg:
> 
> Consider: 
> \`\`\`
> node.roles: \[\]
> \`\`\`
> 
> \`\`\`golang
> input, err := yaml.Unmarshal(\`node.roles: \[\]\`)
> ...
> c, err := NewConfig(\[\]byte(input))
> ...
> var output map\[string\]interface{}
> err = c.Unpack(output)
> ...
> yaml.Marshal(output)
> \`\`\`
> leaves us with:
> 
> \`\`\`
> node.roles: null
> \`\`\`
> 
> We have been hit in the ECK operator a couple of times by this issue:
> \* Kibana Fleet config uses empty arrays to turn off monitoring: https://github.com/elastic/cloud-on-k8s/issues/4853
> \* Elasticsearch uses empty arrays to express the \`coordinating\` node role https://github.com/elastic/cloud-on-k8s/issues/3718
> 
> We are using untyped \`map\[string\]interface{}\` target data structures in the operator for application configuration because it is difficult for us to model each Elastic stack application's configuration completely and for all versions of the Elastic stack that we need to support.

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 22, 2022, 3:19pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/12 "2022-03-22T15:19:10Z")

</div>

Yes Elastic **should fix** this bug.  
Today we need to use workaround for docker compose in section environment (and also didn't find this tips in documentation)  
works for Elasticsearch \<7.17.1

workaround:

1. create docker config for coordinator with legacy settings

```auto
node.master: false
node.data: false
node.ingest: false
node.ml: false
node.remote_cluster_client: false

```

1. map this config in docker compose

configs:  
es-coordination:  
name: es-coordination  
file: /home/Elasticsearch/kickstart\_elk\_cluster/elasticsearch\_coordinator.yml

and You will see in the logs  
`{"type": "deprecation.elasticsearch", "timestamp": "2022-03-22T15:11:18,898Z", "level": "CRITICAL", "component": "o.e.d.n.Node", "cluster.name": "elk_cluster", "node.name": "es_coordination_1", "message": "legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[]]", "key": "legacy role settings", "category": "settings" }`

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 22, 2022, 8:41pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/13 "2022-03-22T20:41:58Z")

</div>

@leandrojmp Can You give an examples how should I define in my docker compose file  
`node.roles: []`

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [March 30, 2022, 4:05pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/14 "2022-03-30T16:05:58Z")

</div>

Any suggest how we can resolve that issue over syntax with `node.roles: []` in docker compose ?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 30, 2022, 4:17pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/15 "2022-03-30T16:17:57Z")

</div>

I'm sorry, I do not use docker and this seems to be a bug.

I would suggest that you open an issue or comment on the one that you shared, however the issue was closed by this [commit](https://github.com/elastic/cloud-on-k8s/pull/5044), so it should be working.

---

<div class="post-metadata">

**Author:** ![sastorsl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sastorsl/32/44972_2.png) [@sastorsl](https://discuss.elastic.co/u/sastorsl)\
**Post date:** [April 21, 2022, 3:09pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/16 "2022-04-21T15:09:10Z")

</div>

> [@INS](#):
>
> `- node.roles= `

probably has to be changed to:

`- node.roles=[]`

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 21, 2022, 3:50pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/17 "2022-04-21T15:50:23Z")

</div>

It was fixed by [#85186] ([Allow yaml values for dynamic node settings by rjernst · Pull Request #85186 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/85186)) (for 8.2.0)  
but till today it hasn't been released ver of 8.2.0

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 19, 2022, 3:51pm UTC](https://discuss.elastic.co/t/change-topic-from-bad-logic-for-allocation-shard-due-to-data-tier-arch-case-with-node-roles-syntax-in-elastic-8-0-0/300168/18 "2022-05-19T15:51:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.