# Allocation awareness - something not right **URL:** \ **Category:** Elasticsearch\ **Created:** [June 6, 2019, 3:09pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616 "2019-06-06T15:09:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 6, 2019, 3:09pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/1 "2019-06-06T15:09:20Z")
I have an 11 node Cluster 3 Master nodes and 8 Data nodes: I have 4 physical machines with 2 Virtual VM's on each, the 2 physical machines are in two different rooms. Data nodes 1,3,5&7 are in room 1 Data nodes 2,4,6&8 are in the other room. The goal is to have the primary shard in 1 room and the replica in the other. **I set on the odd nodes in elasticsearch.yml** cluster.routing.allocation.awareness.attributes: roomid cluster.routing.allocation.awareness.force.roomid.values: 1,2 node.attr.roomid: 1 **I set on the even nodes in elasticsearch.yml** cluster.routing.allocation.awareness.attributes: roomid cluster.routing.allocation.awareness.force.roomid.values: 1,2 node.attr.roomid: 2 when I check the index, I can see that I hace shard 3 in the same room: index-20190531 1 r STARTED 9704310 11.6gb x.x.x.x node1 index-20190531 1 p STARTED 9704310 11.6gb x.x.x.x node8 index-20190531 2 r STARTED 9706200 11.7gb x.x.x.x node2 index-20190531 2 p STARTED 9706200 11.7gb x.x.x.x node3 index-20190531 3 p STARTED 9704938 11.7gb x.x.x.x node2 index-20190531 3 r STARTED 9704938 11.6gb x.x.x.x node6 index-20190531 5 r STARTED 9705567 11.6gb x.x.x.x node7 index-20190531 5 p STARTED 9705567 11.6gb x.x.x.x node4 index-20190531 4 r STARTED 9707267 11.6gb x.x.x.x node7 index-20190531 4 p STARTED 9707267 11.6gb x.x.x.x node6 index-20190531 0 p STARTED 9703313 11.6gb x.x.x.x node5 index-20190531 0 r STARTED 9703313 11.6gb x.x.x.x node8 Is there something I am doing wrong or mis-understanding ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 6, 2019, 4:00pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/2 "2019-06-06T16:00:35Z")
Hi @john275 and welcome! Yes, this does look wrong to me: > [@john275](#): > > ```auto > index-20190531 3 p STARTED 9704938 11.7gb x.x.x.x node2 > index-20190531 3 r STARTED 9704938 11.6gb x.x.x.x node6 > > ``` I'd like to double-check that these settings are applied as you claim on every node. Can you share the full output of the following commands? ```plaintext GET /_nodes/settings?filter_path=nodes.*.settings.node.attr.roomid,nodes.*.name,nodes.*.settings.cluster.routing.allocation.awareness GET /_cluster/settings?filter_path=*.cluster.routing.allocation.awareness ``` ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 6, 2019, 4:47pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/3 "2019-06-06T16:47:23Z")
So the nodes settings yields: {"nodes":{"sZsWrCkwTbmus19eqs1otA":{"name":"node4","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"2"}}}},"lWI-nJwuRlaa-uQ4HSSZ8A":{"name":"node6","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"2"}}}},"a4okbUUvSBSzEXj\_2QD3BA":{"name":"node2","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"2"}}}},"liJIj97NQZuT95-Frnrorg":{"name":"node8","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"2"}}}},"Q9NLbgGXR5m0UqxL\_dUsIw":{"name":"master-a1"},"-Fy\_OEmLRVOlZVnRs1i-9g":{"name":"node1","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"1"}}}},"gGnNGn\_RS0Kv4StnAjZoPQ":{"name":"master-a3"},"tce9HEsSSR6Rv4aM3cNz1g":{"name":"node5","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"1"}}}},"MVmuBaQ3QWalwvHxsz0\_VA":{"name":"node7","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"1"}}}},"xHLpuyCWSk-l9JAM3umK8g":{"name":"master-a2"},"3d-6McBhQGeYUKT8LR7LWw":{"name":"node3","settings":{"cluster":{"routing":{"allocation":{"awareness":{"attributes":"roomid","force":{"roomid":{"values":"1,2"}}}}}},"node":{"attr":{"roomid":"1"}}}}}} and the cluster settings yields {} ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 6, 2019, 5:35pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/4 "2019-06-06T17:35:10Z")
Hmm, ok, that all looks correct to me, thanks. Can you use the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) to ask about the allocation of the problematic shard? You'll need to specify the shard as it's assigned, just not in the right place. Also, what version are you using? ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 6, 2019, 5:55pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/5 "2019-06-06T17:55:18Z")
Thanks for your replies.... I'll have to research using the allocation explain API and reply back here. Version is: "version" : { "number" : "6.6.0", "build\_flavor" : "default", "build\_type" : "deb", "build\_hash" : "a9861f4", "build\_date" : "2019-01-24T11:27:09.439740Z", "build\_snapshot" : false, "lucene\_version" : "7.6.0", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 6, 2019, 6:00pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/6 "2019-06-06T18:00:12Z")
PS I don't think this is limited to just a particular shard, it seems to be a cluster wide issue. Another example index from today: index-20190606 1 r STARTED 6838929 8.9gb x.x.x.x node1 index-20190606 1 p STARTED 6838933 8.7gb x.x.x.x node3 index-20190606 2 r STARTED 6838836 8.9gb x.x.x.x node7 index-20190606 2 p STARTED 6838767 8.9gb x.x.x.x node6 index-20190606 3 p STARTED 6841220 9.7gb x.x.x.x node1 index-20190606 3 r STARTED 6841357 9.4gb x.x.x.x node5 index-20190606 5 r STARTED 6838348 9.9gb x.x.x.x node2 index-20190606 5 p STARTED 6838423 9gb x.x.x.x node8 index-20190606 4 p STARTED 6839791 8.8gb x.x.x.x node4 index-20190606 4 r STARTED 6839791 9.2gb x.x.x.x node8 index-20190606 0 p STARTED 6840745 9.2gb x.x.x.x node7 index-20190606 0 r STARTED 6840678 8.8gb x.x.x.x node3 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 6, 2019, 8:37pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/7 "2019-06-06T20:37:56Z")
Sorry I was on mobile earlier and couldn't check the right syntax. The allocation explain command is this: ```nohighlight GET /_cluster/allocation/explain { "index": "index-20190531", "shard": 3, "primary": false } ``` ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 6, 2019, 10:20pm UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/8 "2019-06-06T22:20:58Z")
> [@john275](#): > > **I set on the odd nodes in elasticsearch.yml** > cluster.routing.allocation.awareness.attributes: roomid > cluster.routing.allocation.awareness.force.roomid.values: room1,room2 > node.attr.roomid: room1 > > **I set on the even nodes in elasticsearch.yml** > cluster.routing.allocation.awareness.attributes: roomid > cluster.routing.allocation.awareness.force.roomid.values: room1,room2 > node.attr.roomid: room2 > [@](#): > > "roomid":{"values":"1,2"} > "attr":{"roomid":"2"} Just to prevent confusion for people reading this thread now or later. What you reported as being your config is not actually how your cluster is configured. Your ids are not room1 and room2 but 1 and 2. I'm not saying that has anything to do with your issue as the output you showed coming from your cluster via GET looks self-consistent. Just not consistent with what you had posted before. Which is confusing, that's all. You probaly just changed the settings in between the posts to use numbers instead of strings. Continue with David's latest question, I don't want to derail your post either or "squirrel!" any of you. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 7, 2019, 6:18am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/9 "2019-06-07T06:18:19Z")
@martinr_ubi makes a good point. Maybe we're not seeing the true information because you consider it to be sensitive? It's ok if you want to redact some things, but please make it clear what, if anything, you've altered. It's all too easy to accidentally obscure the very thing that needs to be adjusted. ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 7, 2019, 6:40am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/10 "2019-06-07T06:40:49Z")
Yes indeed, I have made changes to the IP's, hostnames, index name and room id, Sorry I was not careful enough to change everything carefully and consistently ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 7, 2019, 6:54am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/11 "2019-06-07T06:54:57Z")
Here is the output from GET /\_cluster/allocation/explain { "index": "index-20190531", "shard": 3, "primary": false } {"index":"index-20190531","shard":3,"primary":false,"current\_state":"started","current\_node":{"id":"lWI-nJwuRlaa-uQ4HSSZ8A","name":"node6","transport\_address":"x.x.x.x:9300","attributes":{"ml.machine\_memory":"202844217344","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"2"},"weight\_ranking":1},"can\_remain\_on\_current\_node":"yes","can\_rebalance\_cluster":"yes","can\_rebalance\_to\_other\_node":"no","rebalance\_explanation":"cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance","node\_allocation\_decisions":[{"node\_id":"a4okbUUvSBSzEXj\_2QD3BA","node\_name":"node2","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202844217344","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"2"},"node\_decision":"no","weight\_ranking":1,"deciders":[{"decider":"same\_shard","decision":"NO","explanation":"the shard cannot be allocated to the same node on which a copy of the shard already exists [[index-20190531][3], node[a4okbUUvSBSzEXj\_2QD3BA], [P], s[STARTED], a[id=dmv0skOtReCQ1GZesV6h9w]]"}]},{"node\_id":"-Fy\_OEmLRVOlZVnRs1i-9g","node\_name":"node1","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202843451392","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"1"},"node\_decision":"worse\_balance","weight\_ranking":1},{"node\_id":"3d-6McBhQGeYUKT8LR7LWw","node\_name":"node3","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202843451392","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"1"},"node\_decision":"worse\_balance","weight\_ranking":1},{"node\_id":"MVmuBaQ3QWalwvHxsz0\_VA","node\_name":"node7","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202843451392","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"1"},"node\_decision":"worse\_balance","weight\_ranking":1},{"node\_id":"liJIj97NQZuT95-Frnrorg","node\_name":"node8","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202844217344","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"2"},"node\_decision":"worse\_balance","weight\_ranking":1},{"node\_id":"sZsWrCkwTbmus19eqs1otA","node\_name":"node4","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202844217344","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"2"},"node\_decision":"worse\_balance","weight\_ranking":1},{"node\_id":"tce9HEsSSR6Rv4aM3cNz1g","node\_name":"node5","transport\_address":"x.x.x.x:9300","node\_attributes":{"ml.machine\_memory":"202843451392","ml.max\_open\_jobs":"20","xpack.installed":"true","ml.enabled":"true","roomid":"1"},"node\_decision":"worse\_balance","weight\_ranking":1}]} I'll check the other index from 06/06 and report back here as that cluster has been stable since that index was created. ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 7, 2019, 7:12am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/12 "2019-06-07T07:12:29Z")
The other index 06/06 yields the same narrative for all of the shards: "rebalance\_explanation": "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance", ``` "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[index-20190606][5], node[liJIj97NQZuT95-Frnrorg], [P], s[STARTED], a[id=EvLUyDqLSE-P8_PdSDhHZQ]]" ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 7, 2019, 9:30am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/13 "2019-06-07T09:30:43Z")
The telling line is this one: `"can_remain_on_current_node": "yes",` which tells us that the awareness allocator (and all other allocation decoders) are happy. I'm mystified. I've traced through the code from 6.6.0 and can't see how this could be happening, if your config is exactly as you describe. I am concerned that we're losing something vital when you're obscuring the room attributes. ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 7, 2019, 10:29am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/14 "2019-06-07T10:29:29Z")
mmm, now that I read from the top wasn't it under our nose the whole time: > [@](#): > > **I set on the odd nodes in elasticsearch.yml** > **I set on the even nodes in elasticsearch.yml** > "Q9NLbgGXR5m0UqxL\_dUsIw":{"name":"master-a1"} > "gGnNGn\_RS0Kv4StnAjZoPQ":{"name":"master-a3"} > "xHLpuyCWSk-l9JAM3umK8g":{"name":"master-a2"} You're not _also_ putting the settings on the master nodes? ```auto cluster.routing.allocation.awareness.attributes: roomid cluster.routing.allocation.awareness.force.roomid.values: 1,2 ``` Which I believe is critical. I guess you should do that. ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 7, 2019, 10:33am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/15 "2019-06-07T10:33:09Z")
I'll add the original config here, but will remove it once you have validated, I have not mangled anything. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 7, 2019, 10:34am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/16 "2019-06-07T10:34:56Z")
Ok I've taken a copy. ---
**Author:** ![john275](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@john275](https://discuss.elastic.co/u/john275)\ **Post date:** [June 7, 2019, 10:38am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/17 "2019-06-07T10:38:50Z")
Actually, you might have it Martin, I recall pondering that at the time I was setting it and it dropped from my mind. So I should add: cluster.routing.allocation.awareness.attributes: roomid cluster.routing.allocation.awareness.force.roomid.values: 1,2 #but with my true roomid values I'll give that a try next week, unless Dave finds anything more relevant ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 7, 2019, 10:46am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/18 "2019-06-07T10:46:32Z")
> Now, we need to set up _shard allocation awareness_ by telling Elasticsearch which attributes to use. This can be configured in the `elasticsearch.yml` file on **all** master-eligible nodes, or it can be set (and changed) with the [cluster-update-settings](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/cluster-update-settings.html) API. > **[Shard Allocation Awareness | Elasticsearch Guide \[6.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/allocation-awareness.html#allocation-awareness)** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 7, 2019, 10:46am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/19 "2019-06-07T10:46:55Z")
Oh of course. It's the master node that is making the decision about where to allocate things, so it needs to know that allocation awareness is enabled for this attribute. When I tried to reproduce, I had 8 nodes that were all master-eligible and data nodes, but of course that's not what's going on here. In fact there's no need to add the `cluster.routing.allocation.awareness.*` settings on the data nodes at all, they just need the `node.attr.roomid` settings. ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 7, 2019, 10:59am UTC](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616/20 "2019-06-07T10:59:07Z")
I need to look into making MR against the doc 🙂 That'd be cool. The doc could use more clarity here, I just read it all again and it doesn't specifically says node attributes goes on the data nodes and the awareness settings goes in the masters eligible. ( It does say it for the awareness.attributes, but not VS the node.attr ) And more importantly doesn't repeat any of that, lower, in the "Forced Awareness" section of the page. So in the end the force.$attrib.values setting is never referenced in terms of where it goes. Where each settings go should be said or repeated outside the normal narrative to make it special and important is what I'm saying. For someone new to Elasticsearch just learning about node types, or just reading the section about Force Awareness at a later time because he read the first section in the past... That leads to the current forum thread 🧐 Let's be clear, that doc is not that bad, but I'm pretty sure @john275 had to read it at some point or else he wouldn't even know about awareness. So somehow he missed it. Maybe exactly because he read only the forced awareness section? Just thinking aloud. [Next page](https://discuss.elastic.co/t/allocation-awareness-something-not-right/184616.md?page=2)