# What is the \`index.number\_of\_routing\_shards\` setting? How can I calculate it based on the number of shards?

**URL:** <https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540>\
**Category:** Elasticsearch\
**Created:** [February 17, 2022, 7:45pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540 "2022-02-17T19:45:41Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 17, 2022, 7:45pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/1 "2022-02-17T19:45:41Z")

</div>

I'm aware of a similar question to this topic. Namely [What are "number\_of\_routing\_shards"? - #2 by steffens](https://discuss.elastic.co/t/what-are-number-of-routing-shards/120506/2). But it does not answer my question or help me understand the `index.number_of_routing_shards` setting.

I'm hoping someone can help me _grok_ the [index.number\_of\_routing\_shards](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-routing-shards) setting. I've come across it in relation to [\_routing](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html) where it is used to select a shard. e.g.

> routing\_factor = **num\_routing\_shards** / num\_primary\_shards  
> shard\_num = (hash(\_routing) % **num\_routing\_shards** ) / routing\_factor

I don't understand what this value would be in practice. A multiple of the primary shard number, I get. But I don't know how I'd calculate it. For example, in an index with the `num_primary_shards` set to `90`, how can I deduce the `number_of_routing_shards`?

To provide context, we're evaluating if we should adopt a custom **\_routing** rule. I'm introducing metrics that'll calculate how balanced / unbalanced a custom routing would make our Elasticsearch indexes. i.e. Would we route more traffic to some shards versus others. To this end I need to be able to determine which shard will be used. Looking at the documentation the formula seems relatively simple, but I'm struggling to understand `number_of_routing_shards`, which is central to it. Initially I thought it might just be the total number of shards. But that does not seem to be the case.

BTW if you've an alternative approach or know a utility I can call to calculate the selected shard that too would be of interest.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 17, 2022, 7:58pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/2 "2022-02-17T19:58:49Z")

</div>

I think a custom routing rule based on this knowledge would end up being dangerously brittle, but the docs you link pretty much say how to calculate its default:

> This setting’s default value depends on the number of primary shards in the index. The default is designed to allow you to split by factors of 2 up to a maximum of 1024 shards.

In other words you take `num_primary_shards` and multiply it by 2 repeatedly until you get a number between 513 and 1024. For 90 it's `90*2*2*2=720`.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 17, 2022, 9:17pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/3 "2022-02-17T21:17:46Z")

</div>

I'm content to take your word on `720`. To be honest I can't claim I would have come to that conclusion by myself. Nor did the ~10 engineers I asked to look at the documentation. Perhaps it's obvious to ES team member but harder for someone outside ES to understand. An enhancement to the documents might be worthwhile.

> I think a custom routing rule based on this knowledge would end up being dangerously brittle

I'd be interested to hear more on this. The brittleness I'm considering is if a shard (and it's two replicas) goes offline. Then anything being read or written to it, by the routing rule, is in trouble. Are there any other concerns / dangers?

The temptation is that, for 50% of our traffic, we can drop the search request cost from 90 shards to a single shard. Which would help with our CPU usage.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 18, 2022, 11:21am UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/4 "2022-02-18T11:21:25Z")

</div>

> [@Shane\_Gannon](#):
>
> An enhancement to the documents might be worthwhile.

It's deliberately obscure because this is very much an implementation detail on which you shouldn't rely.

> [@Shane\_Gannon](#):
>
> I'd be interested to hear more on this. The brittleness I'm considering...

I meant that if you design a routing scheme based on intimate knowledge of how each routing value maps to the individual shards, perhaps carefully choosing routing values so that each shard remains roughly equal in size, then you are relying on these implementation details not changing in future. If they did change (or say you changed the number of shards in your index) then you could end up with something very unbalanced in its place.

> [@Shane\_Gannon](#):
>
> we can drop the search request cost from 90 shards to a single shard. Which would help with our CPU usage.

A single shard which contains all the matching docs so has to work correspondingly harder tho, so it's not a guaranteed win.

How large are your shards out of interest?

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 22, 2022, 10:47am UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/5 "2022-02-22T10:47:01Z")

</div>

> [@DavidTurner](#):
>
> It's deliberately obscure because this is very much an implementation detail on which you shouldn't rely.

That makes sense. Helps explain why I found it hard to understand.

> [@DavidTurner](#):
>
> then you are relying on these implementation details not changing in future

I see what you mean. In terms of the `number_of_routing_shards` setting that's not a big worry. The plan is for this calculation to be temporary. We'll use it to generate metrics to tell us how stable the solution would be. With this information we can come to a go / no go decision. After which the code will be removed. As we're targeting a specific version of ES7, and have no plans to upgrade within a near period of time, we should _get away with it_. I did notice this logic has changed in ES8.

Our data model is made up of documents, groups, and organisations. Currently _\_routing_ is based on the document's id. The metrics I'm putting in place will simulation the effect of switching _\_routing_ over to the group and organisation ids. After which we'll weigh the pros and cons.

> [@DavidTurner](#):
>
> A single shard which contains all the matching docs so has to work correspondingly harder tho, so it's not a guaranteed win.

This is true but we think, with the group id, the balance should be roughly consistent. With the organisation id we're more at risk from large orgs. Which is why the group id is the favourite.

> [@DavidTurner](#):
>
> How large are your shards out of interest?

The current index's shards are ~30 GB. The shards in the previous indexes reached ~50-57 GB before a new index was cut. A new index is cut once every ~6 weeks.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [March 3, 2022, 12:27pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/6 "2022-03-03T12:27:53Z")

</div>

The code I used to simulate \_routing is

```auto
int partitionOffset = 0;
int routingFactor = properties.getEsRoutingSimulationNumberOfRoutingShards() / properties.getEsRoutingSimulationNumberOfShards();
int hash = Murmur3HashFunction.hash(effectiveRouting) + partitionOffset;
return Math.floorMod(hash, properties.getEsRoutingSimulationNumberOfRoutingShards()) / routingFactor;

```

where `getEsRoutingSimulationNumberOfShards()` is `30` and `getEsRoutingSimulationNumberOfRoutingShards()` is set to `960`.

Looking at the simulation results for the doc \_id, rather than the group or org id, the results look mostly ok. But **one** shard seems to be getting more documents routed towards it. i.e. It gets 4% of the overall traffic versus the other shards receiving 3%. Checking Elasticsearch directly via it's _\_cat/shards_ API its more balanced. i.e. All the shards seem to be on 3%.

That makes me think that either

- I've a mistake, or missed something, in the formula above or property values
- Or Elasticsearch does a little more load balancing under the covers

Any chance the second point is possible? Especially when there is no custom [\_routing](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html) rule defined.

 ![simulation](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fcd79119182f6e44e2a124ad3b6e9046269eacfe.jpeg)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 3, 2022, 1:24pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/7 "2022-03-03T13:24:52Z")

</div>

I can't reproduce this. I tried this test:

```java
    public void testDistribution() {
        var numRoutingShards = 960;
        var numShards = 30;
        var routingFactor = numRoutingShards / numShards;
        var frequencies = new int[numShards];
        var iterations = 100_000_000;
        for (int i = 0; i < iterations; i++) {
            frequencies[Math.floorMod(Murmur3HashFunction.hash(UUIDs.base64UUID()), numRoutingShards) / routingFactor] += 1;
        }
        var expected = iterations / numShards;
        for (int i = 0; i < frequencies.length; i++) {
            logger.info("frequencies[{}] = {} (deviation from mean {})", i, frequencies[i], frequencies[i] - expected);
        }
        var minFreq = Arrays.stream(frequencies).min().orElseThrow();
        var maxFreq = Arrays.stream(frequencies).max().orElseThrow();
        logger.info("min {} max {} range {}", minFreq, maxFreq, maxFreq-minFreq);
    }

```

Here's a typical output:

```nohighlight
[2022-03-03T18:22:30,551][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] before test
[2022-03-03T18:22:42,529][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[0] = 3333122 (deviation from mean -211)
[2022-03-03T18:22:42,529][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[1] = 3333152 (deviation from mean -181)
[2022-03-03T18:22:42,529][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[2] = 3333617 (deviation from mean 284)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[3] = 3332770 (deviation from mean -563)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[4] = 3332795 (deviation from mean -538)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[5] = 3334148 (deviation from mean 815)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[6] = 3333634 (deviation from mean 301)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[7] = 3330884 (deviation from mean -2449)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[8] = 3335396 (deviation from mean 2063)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[9] = 3330848 (deviation from mean -2485)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[10] = 3330694 (deviation from mean -2639)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[11] = 3335203 (deviation from mean 1870)
[2022-03-03T18:22:42,530][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[12] = 3336015 (deviation from mean 2682)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[13] = 3332726 (deviation from mean -607)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[14] = 3329691 (deviation from mean -3642)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[15] = 3333681 (deviation from mean 348)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[16] = 3332674 (deviation from mean -659)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[17] = 3337441 (deviation from mean 4108)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[18] = 3335784 (deviation from mean 2451)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[19] = 3331326 (deviation from mean -2007)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[20] = 3332416 (deviation from mean -917)
[2022-03-03T18:22:42,531][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[21] = 3331359 (deviation from mean -1974)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[22] = 3334882 (deviation from mean 1549)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[23] = 3333240 (deviation from mean -93)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[24] = 3333283 (deviation from mean -50)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[25] = 3332731 (deviation from mean -602)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[26] = 3333295 (deviation from mean -38)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[27] = 3334274 (deviation from mean 941)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[28] = 3336348 (deviation from mean 3015)
[2022-03-03T18:22:42,532][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] frequencies[29] = 3332571 (deviation from mean -762)
[2022-03-03T18:22:42,533][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] min 3329691 max 3337441 range 7750
[2022-03-03T18:22:42,551][INFO][o.e.c.r.o.h.m.Murmur3HashFunctionTests] [testDistribution] after test

```

I'm going to guess that something has introduced some skew into your doc IDs.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [March 3, 2022, 3:27pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/8 "2022-03-03T15:27:41Z")

</div>

> [@DavidTurner](#):
>
> I'm going to guess that something has introduced some skew into your doc IDs.

That was my first suspicion. It makes sense, though the UUIDs (aka `effectiveRouting`) are fairly straight forward.

What makes me concerned is that this is not replicated in the shard's document count returned by the [/\_cat/shards](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html) API. That means, unless Elasticsearch 7 has extra routing logic to help balance the shards, then something is wrong in the formula / my simulation code.

I'll try focus on that area.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 31, 2022, 3:28pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540/9 "2022-03-31T15:28:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.