# Skewed shards on index even with enough disk space in all physical nodes
**URL:** https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974
**Category:** Elasticsearch
**Created:** [October 23, 2019, 10:55pm UTC](https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974 "2019-10-23T22:55:07Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@viniciof](https://discuss.elastic.co/u/viniciof)
#### Post date: [October 23, 2019, 10:55pm UTC](https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974/1 "2019-10-23T22:55:08Z")
Hi everyone,
I have an index (shown below) which is not evenly distributed accross my 10 data nodes cluster.
Each node in the cluster has at least 1.1 TB free, only the one called arm-lc-004 has ~700GB free only but it's also quite enough.
Where should I look at to try to diagnose the root cause behind these skewed shards ?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 24, 2019, 7:07am UTC](https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974/2 "2019-10-24T07:07:59Z")
> [@viniciof](#):
>
> Where should I look at to try to diagnose the root cause behind these skewed shards ?
Can you share the output of `GET _cat/shards`? Use [https://gist.github.com/](https://gist.github.com/) if it doesn't fit here.
---
### Author:  [@viniciof](https://discuss.elastic.co/u/viniciof)
#### Post date: [October 24, 2019, 3:14pm UTC](https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974/3 "2019-10-24T15:14:46Z")
Hi Dave,
Output is [here](https://gist.github.com/vinicioflores/fdf0e685c64ace776637eda2c29bbcb8)
Appreciate any inputs,
Regards,
Vinicio
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 24, 2019, 3:48pm UTC](https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974/4 "2019-10-24T15:48:43Z")
It looks like Elasticsearch is aware of the imbalance and is moving shards to address it. This rebalancing process takes time.
```nohighlight
$ grep es-shards.txt -e RELOCATING
daas-arm-prod-users-2019-10 4 r RELOCATING 13259050 59.4gb 10.187.72.6 arm-lc-004_data -> 10.187.72.4 Be-sYmy7TJquJWCAmZ2aSA arm-lc-002_data
daas-arm-prod-users-2019-09 3 p RELOCATING 16503328 69.7gb 10.187.72.6 arm-lc-004_data -> 10.187.72.4 Be-sYmy7TJquJWCAmZ2aSA arm-lc-002_data
```
Some of your indices have far too many shards that are far too small. E.g. `daas-arm-int-dataflow-2019-10` has 10 shards all smaller than 70MB. Indices like this should have one shard.
You have some excessively tiny daily indices too, e.g. `kafka-metrics-*` and `jmx-*`. These would be better as one-shard monthly indices.
Other shards look to be time-based but are surprisingly large. E.g. `daas-arm-prod-users-2019-10` has 20 shards all in the region of 60GB. 60GB shards are fine, but why put 20 of them into one index? Would you be able to have fewer shards at once and use [rollover](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/indices-rollover-index.html) to start a new index when the old one gets too big?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 21, 2019, 3:48pm UTC](https://discuss.elastic.co/t/skewed-shards-on-index-even-with-enough-disk-space-in-all-physical-nodes/204974/5 "2019-11-21T15:48:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.