# Shard allocation based on shard size
**URL:** https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817
**Category:** Elasticsearch
**Created:** [December 7, 2020, 8:19am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817 "2020-12-07T08:19:44Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 7, 2020, 8:19am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/1 "2020-12-07T08:19:45Z")
Hi,
I was wondering if there is an option to include shard size in the shard allocation decision process.
The [Disk-based shard allocation settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation) and the number of shards/node works, and most of or shards are 50G in size, but sometimes a few 2-3G shards get in the mix. When this happens, ES only looks at the number of shards/node, not if the total shard size looks equal.
Is this something ES can take into account? Or will I need to write my own rebalancing logic for this kind of behaviour?
Thanks!
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 7, 2020, 8:34am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/2 "2020-12-07T08:34:03Z")
It currently works on shard count only, this can sometimes lead to lumpy disk use across nodes.
Is it causing issues?
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 7, 2020, 8:37am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/3 "2020-12-07T08:37:07Z")
Well yes, I'm trying to keep as much data in ES as possible, but when nodes of a specific type (hot/warm/cold) are not evenly distributed, I waste a lot of disk space 🙂.
I guess I'll need to write a rebalancer based on shard sizes which runs every once and a while.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 7, 2020, 8:39am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/4 "2020-12-07T08:39:03Z")
How much disk space are you talking here? Are you able to share the output from `_cat/allocation?v`?
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 7, 2020, 9:31am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/5 "2020-12-07T09:31:57Z")
Part of the node list:
```
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
106 4.3tb 4.3tb 639.6gb 4.9tb 87 elkdatac001 x.y.4.13 elkdatac001
107 4.7tb 4.7tb 216.5gb 4.9tb 95 elkdatac002 x.y.4.14 elkdatac002
107 4.5tb 4.5tb 410gb 4.9tb 91 elkdatac003 x.y.4.15 elkdatac003
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 7, 2020, 11:22am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/6 "2020-12-07T11:22:57Z")
Doesn't look like much wasted space to me 🙂
Since you have quite large disks you might like to consider configuring the [disk watermarks](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-cluster.html#disk-based-shard-allocation) differently. The high watermark default of 90% means that Elasticsearch tries to keep 500GB free on each 5TB disk. That's not totally silly, there is some belief that filesystem performance drops once disks get too full, but if you would rather run closer to the wire then that's your call.
Elasticsearch aims to keep nodes under the high watermark but will only move shards between nodes when necessary. This means that "evening out" disk usage is deliberately avoided but it doesn't mean that any space is wasted, even if your shards have rather different sizes.
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 7, 2020, 12:04pm UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/7 "2020-12-07T12:04:41Z")
Yes, I know 🙂 , I already changed the high watermark, but as you can see in the example, at least 8 shards (of 50G) could be added in elkdatac001 when total shard size would be included in the rebalancing.
Thanks for the answers!
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 7, 2020, 12:24pm UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/8 "2020-12-07T12:24:04Z")
> [@coudenysj](#):
>
> but as you can see in the example, at least 8 shards (of 50G) could be added in elkdatac001 when total shard size would be included in the rebalancing.
Sure, but does Elasticsearch have a good reason to move any shards onto this node? If so, what is it?
If for instance `elkdatac002` were above its high watermark then Elasticsearch would indeed move shards across to balance things out.
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 9, 2020, 9:12am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/9 "2020-12-09T09:12:35Z")
Well, the good reason would be "the disks are not filled evenly" 🙂.
The current example I have now, are the disks on my hot nodes:
```
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
84 749.4gb 752.1gb 93.3gb 845.4gb 88 elkdatah001 x.34.4.7 elkdatah001
88 643.6gb 645.9gb 199.5gb 845.4gb 76 elkdatah002 x.34.4.8 elkdatah002
87 664.3gb 665.2gb 180.2gb 845.4gb 78 elkdatah003 x.34.4.9 elkdatah003
87 690.9gb 691.9gb 153.5gb 845.4gb 81 elkdatah004 x.34.4.240 elkdatah004
87 643.7gb 644.9gb 200.5gb 845.4gb 76 elkdatah005 x.34.4.241 elkdatah005
87 704.6gb 705.9gb 129.7gb 835.6gb 84 elkdatah006 x.34.4.242 elkdatah006
```
The first one has a lot of large shards, but as you can see, ES does not allocate any new incides to this node, but the disk numbers are still a problem.
That is why I think it is strange the total disk size is not taken into account in the relocating logic, but it is also strange I'm the only person running into this problem 😃.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 9, 2020, 9:32am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/10 "2020-12-09T09:32:18Z")
> [@coudenysj](#):
>
> Well, the good reason would be "the disks are not filled evenly" 🙂.
That in itself isn't a good reason to move shards around. Moving a shard is an expensive operation, it's not worth doing simply for the sake of tidiness.
> [@coudenysj](#):
>
> the disk numbers are still a problem.
This is what I'm not understanding. I see that the numbers aren't equal, but I don't see why this is a problem. How would your life be better if the numbers were closer together? Is there some operational issue that this unevenness is causing?
> [@coudenysj](#):
>
> That is why I think it is strange the total disk size is not taken into account in the relocating logic, but it is also strange I'm the only person running into this problem 😃.
You're not the only person to experience this confusion, and we recently expanded the [docs on this subject](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation) for that reason. Disk space absolutely is taken into account when relocating shards, but that doesn't imply we aim for equal disk usage across nodes. That goal is expensive and unnecessary.
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 21, 2020, 8:06am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/11 "2020-12-21T08:06:12Z")
Hi @DavidTurner, thanks for the answers.
After thinking about it for a while, I'll just gonna try to reconfigure our own host monitoring. As Elastic indeed has DiskUsage checks, the hosts will probably eventually become evenly distributed.
The problem we see is that in busy days, the apache filebeat logs grow a lot, so we need to have some buffer, but we want it as small as possible.
But I'll start by removing our own disk checks, thanks again for the feedback!
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 21, 2020, 8:27am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/12 "2020-12-21T08:27:40Z")
> [@coudenysj](#):
>
> The problem we see is that in busy days, the apache filebeat logs grow a lot, so we need to have some buffer, but we want it as small as possible.
I see - that is a better reason for relocating shards 🙂
The usual solution is to set the gap between the low and high watermarks to be larger than the typical size of the day's indices on each node, and the gap between the high and flood-stage watermarks to be large enough to allow time to mitigate any overage before disks fill up. This largely works in practice, but it's not completely ideal.
---
### Author:  [@coudenysj](https://discuss.elastic.co/u/coudenysj)
#### Post date: [December 21, 2020, 8:32am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/13 "2020-12-21T08:32:11Z")
Well, I basically have (had) two problems, the monitoring system (which I disabled for the ES data mount, because the watermark system works well for this) and the busy days.
Because I use ILM, the watermark system is not really helping me. When the hot nodes are full, I need to manually change the ILM config to make sure the indices get allocated to warm or cold nodes. But that is something I will look at in the future. For now a bigger buffer wil do.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 21, 2020, 8:38am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/14 "2020-12-21T08:38:30Z")
I see, in which case it sounds like you might be looking for [#47764](https://github.com/elastic/elasticsearch/issues/47764). Please feel free to leave a comment (even just a +1) to let us know you'd like us to work on it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 18, 2021, 8:38am UTC](https://discuss.elastic.co/t/shard-allocation-based-on-shard-size/257817/15 "2021-01-18T08:38:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.