# ES allocates primary shards on the same data node **URL:** \ **Category:** Elasticsearch\ **Created:** [July 10, 2018, 10:44am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326 "2018-07-10T10:44:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 10, 2018, 10:44am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/1 "2018-07-10T10:44:27Z")
Hi, i am creating an index with 10 primary shards and 0 replicas, however ES keeps creating the shards on the same data node. i tried to set cluster.routing.allocation.balance.index to 0.75 but this seem to have no effect. in fact, I've noticed that after changing this setting, shards started to reallocate, but of indexes that their shards are already distributed among different data nodes (which was a surprise). i looked at this post: [https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-total-shards.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-total-shards.html) and been wondering (haven't tried that yet) - why my configuration does not have the effect i expect it to have while the suggested setting seem to tackle the exact same issue i am facing? Why ES insists on creating primaries shards of the same index on the same data node? How can i make ES allocate the shards on different data nodes (even at the cost of failing the creation of index)? thanks, Ofer ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [July 10, 2018, 9:28pm UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/2 "2018-07-10T21:28:50Z")
What version of Elasticsearch are you using and crucially, how many nodes are in the cluster? ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 2:54am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/3 "2018-07-11T02:54:55Z")
Hi, I am using ES 5.6, and my cluster has 22 data nodes. ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 2:56am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/4 "2018-07-11T02:56:30Z")
Adding more info, there are 3 master nodes and 20 coordinating nodes. An overall of 10k shards, the cluster is always in green state. ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [July 11, 2018, 4:02am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/5 "2018-07-11T04:02:00Z")
Ok, so in total, 45 node cluster running Elasticsearch 5.6.x _(which patch version - 5.6.10?)_: - 3 dedicated master nodes - 22 data nodes - 20 coordinating (i.e. not data, not master) nodes And with a 10 primary shard index, what distribution of the primary shards are you seeing? ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 4:16am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/6 "2018-07-11T04:16:45Z")
Ill get back to you about that patch version. All 10 primary shards are created immediately on the same data node. I tried to delete the index and re-create it, they still created on a single data node, and oddly always on that same one. ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [July 11, 2018, 4:39am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/7 "2018-07-11T04:39:31Z")
What does the log file show on that data node for when the index is created? If you enumerate the shard ids and execute [Cluster Allocation Explain API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-allocation-explain.html), what is returned for each shard? ```json GET /_cluster/allocation/explain?include_yes_decisions=true { "index": "", "shard": 0, "primary": true } ``` ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 5:48am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/8 "2018-07-11T05:48:23Z")
hi, version details: "version" : { "number" : "5.6.3", "build\_hash" : "1a2f265", "build\_date" : "2017-10-06T20:33:39.012Z", "build\_snapshot" : false, "lucene\_version" : "6.6.1" } when i enumerate through the shards with the explain api, all shards return the following rebalance\_explanation: "rebalance\_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance" from the data node log, in the 5 minutes time frame before the index was created, i see the following message repeating multiple times - on a different index: [2018-07-10T10:30:00,877][DEBUG][o.e.a.b.TransportShardBulkAction] [iapp707-data] [mount\_search-2018.07.10\_0700][4] failed to execute bulk item (index) BulkShardRequest [[mount\_search-2018.07.10\_0700][4]] containing [37] requests ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2018, 5:56am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/9 "2018-07-11T05:56:43Z")
Can you verify that all nodes are running exactly the same version, by running `GET /_cat/nodes?h=id,ip,v,m` ? ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 6:13am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/10 "2018-07-11T06:13:08Z")
yes, all nodes are running 5.6.3: curl --silent "my\_ip/\_cat/nodes?h=v" | sort | uniq 5.6.3 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2018, 6:16am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/11 "2018-07-11T06:16:52Z")
Have you tried using [the total shards per node index setting](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/allocation-total-shards.html)? ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 6:19am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/12 "2018-07-11T06:19:53Z")
i did not tried that setting. i tried to change cluster.routing.allocation.balance.index to 0.75, expecting the shards to start relocate, but that never happened. as i wrote above, i am confused as why my setting change did not have any effect (or at least the effect i expected it), and what is the difference to your suggestion? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2018, 6:20am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/13 "2018-07-11T06:20:48Z")
Cluster settings take all indices into account while the one I linked to is per index. ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 6:25am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/14 "2018-07-11T06:25:13Z")
so if most of the indices are distributed correctly, setting cluster.routing.allocation.balance.index to higher value may not have the expected effect because overall it will still not cross allocation.balance.threshold? and you suggested setting is more 'aggressive', meaning there is no threshold involved, just making sure the shards are distributed? also - is this setting only enforced at index creation time or also will reallocate existing indices? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2018, 6:28am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/15 "2018-07-11T06:28:26Z")
It is a dynamic setting, so even though it probably would be good to set through an index template, I believe it should take effect and cause a rebalancing even if applied at a later stage. I have however not used it in a long time so am not entirely sure. Best way to find out is probably to try. ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 6:30am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/16 "2018-07-11T06:30:31Z")
i will give it try and update soon - thanks ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 6:40am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/17 "2018-07-11T06:40:36Z")
it worked - all shards are now distributed on different nodes. do you recommend to apply this setting to all indices, or 'as needed'? i am referring to note at the bottom of the documentation page: _"These settings impose a hard limit which can result in some shards not being allocated. Use with caution."_ ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2018, 6:44am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/18 "2018-07-11T06:44:47Z")
If you use this by default and lose a few data nodes so that all primary and replica shards can not be allocated to distinct hosts I assume the index would go to a yellow state (red if all primaries could not be all allocated). You are in a better position to judge what impact this would have on your use case and how likely it is to happen. ---
**Author:** ![Oferkes](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@Oferkes](https://discuss.elastic.co/u/Oferkes)\ **Post date:** [July 11, 2018, 6:54am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/19 "2018-07-11T06:54:56Z")
Christian/forloop, thanks for helping with this issue! Ofer ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2018, 6:54am UTC](https://discuss.elastic.co/t/es-allocates-primary-shards-on-the-same-data-node/139326/20 "2018-08-08T06:54:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.