# Shard Reallocation While Indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [August 6, 2025, 4:58am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779 "2025-08-06T04:58:28Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![nchalise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchalise/32/130399_2.png) [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Post date:** [August 6, 2025, 4:58am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/1 "2025-08-06T04:58:28Z")
We have 20 indexes with 490 shards. The shards size are vary from 35GB to 180GB and the ES cluster size is 35 data nodes. Each data node has 5 disk each disk has size 500GB. While indexing we noticed that shards are starting reallocation. ES version used was 9.0 and total size of these 20 indexes are 55TB. Other ES setting are default one. Here, My query are: Why shard reallocation were performed at the time of indexing?Is it due to rebalancing the disk space? Is data were lost if we perform indexing at the time of shard reallocation? How to avoid shard reallocation at the time of data indexing? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [August 6, 2025, 7:50am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/2 "2025-08-06T07:50:34Z")
There was a somewhat similar recent thread [here](https://discuss.elastic.co/t/is-there-a-way-to-determine-what-triggers-shard-movement/380047) which contains some useful diagnostic tips/info, as well as linking to other threads. As in that thread, might be useful to share output of ```auto GET /_cat/nodes?v&h=name,role,disk.used_percent,disk.used,disk.avail&s=role ``` > [@nchalise](#): > > How to avoid shard reallocation at the time of data indexing? If at any instant there was no indexing ongoing how can the cluster know there will be no indexing in the next instant/second/minute/whatever? I guess if your indexing only happens at very specific and predictable times, you could code something, but ... seems wrong. Is the reallocation actually causing you / your clients some actual issue? Or is it that it's just unexpected. And, is your cluster stable, nodes are not semi-frequently leaving/re-joining the cluster? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 6, 2025, 7:59am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/3 "2025-08-06T07:59:07Z")
> [@nchalise](#): > > Is data were lost if we perform indexing at the time of shard reallocation? No. > [@nchalise](#): > > How to avoid shard reallocation at the time of data indexing? There's no need to avoid this. ---
**Author:** ![nchalise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchalise/32/130399_2.png) [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Post date:** [August 6, 2025, 8:40am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/4 "2025-08-06T08:40:25Z")
Thanks for the response. Here, we also observed disk water mark issue along with shard reallocation.So,If node is in read-only mode (e.g., due to disk watermark), writes will silently fail without any notification while data indexing? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 6, 2025, 8:46am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/5 "2025-08-06T08:46:55Z")
Are you using the default watermark settings or have you customised this in any way? ---
**Author:** ![nchalise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchalise/32/130399_2.png) [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Post date:** [August 6, 2025, 8:50am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/6 "2025-08-06T08:50:03Z")
Using default watermark settings. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 6, 2025, 9:07am UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/7 "2025-08-06T09:07:45Z")
> [@nchalise](#): > > we also observed disk water mark issue along with shard reallocation What did you observe exactly? Did a node exceed the high watermark briefly? If so, this is [described in the docs](https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#disk-based-shard-allocation) as normal behaviour: > It is normal for nodes to temporarily exceed the high watermark from time to time. ---
**Author:** ![nchalise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchalise/32/130399_2.png) [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Post date:** [August 6, 2025, 12:28pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/8 "2025-08-06T12:28:58Z")
Each node in our cluster is configured with five disks per node, each sized at 500GB—giving a total of 2.5TB per node. With Elasticsearch’s default disk watermarks enabled, does the system evaluate disk usage individually per disk, or does it consider the combined disk space across the node when triggering watermark-related error? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [August 6, 2025, 12:54pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/9 "2025-08-06T12:54:00Z")
the 5 disk's partitions are each mounted at a different mount points, and you are using Multiple Data Paths ? i.e. you have an entry like ```auto path.data: /mnt/data1,/mnt/data2,/mnt/data3,/mnt/data4,/mnt/data5 ``` in your nodes' elasticsearch.yml ? Can you also share output of ```auto GET /_cat/nodes?v&h=name,role,disk.used_percent,disk.used,disk.avail&s=role ``` and ```auto GET /_cluster/settings?include_defaults=true&filter_path= **.disk.watermark.** ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 6, 2025, 1:04pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/10 "2025-08-06T13:04:47Z")
Which version of Elasticsearch are you using? ---
**Author:** ![nchalise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchalise/32/130399_2.png) [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Post date:** [August 6, 2025, 3:10pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/11 "2025-08-06T15:10:04Z")
Yes, it is like path.data: /mnt/data1,/mnt/data2,/mnt/data3,/mnt/data4,/mnt/data5 Cluster has been terminated so no other data for \_cluster and \_cat available. ---
**Author:** ![nchalise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nchalise/32/130399_2.png) [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Post date:** [August 6, 2025, 3:10pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/12 "2025-08-06T15:10:55Z")
Which version of Elasticsearch are you using? 9.0.0 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 6, 2025, 3:18pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/13 "2025-08-06T15:18:03Z")
According to [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/important-settings.html) specifying multiple datra paths the way you have done is deprecated: > Elasticsearch offers a deprecated setting that allows you to specify multiple paths in `path.data` . To learn about this setting, and how to migrate away from it, refer to [Multiple data paths](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/path-settings-overview.html#multiple-data-paths). As far as I know Elasticsearch is unable to move shards between data paths on the same host, which may complicate reallocation. I would recommend changing this as outlined in the docs I linked to. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [August 6, 2025, 3:38pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/14 "2025-08-06T15:38:27Z")
> [@nchalise](#): > > Yes, it is like path.data: /mnt/data1,/mnt/data2,/mnt/data3,/mnt/data4,/mnt/data5 OK, thank you. That mechanism is being deprecated. See the [docs](https://www.elastic.co/docs/reference/elasticsearch/index-settings/path). For a bunch of reasons you would likely be better served by using LVM tools to collect your 5-disks-per-node into a single filesystem and let the operating system/filesystem manage the space. If your system was working this is a lengthy process, 35 nodes, but doable. Personally, old school, but 5x35 = 165 disks is a bit too many for me, so I'd try to use some sort of RAID, software RAID if necessary. But > [@nchalise](#): > > Cluster has been terminated so no other data for \_cluster and \_cat available. Err, not sure how to interpret "terminated". You mean crashed, not currently working, unable to get working, ... ? You are looking for assistance to get it working again ? ---
**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\ **Post date:** [August 6, 2025, 6:29pm UTC](https://discuss.elastic.co/t/shard-reallocation-while-indexing/380779/15 "2025-08-06T18:29:30Z")
shard size of 180GB? Isn't that way too large? In addition, your disk size is only 500GB each. One such shard would occupy 35% of the storage already. It would probably throw off any rebalancing algorithm. I believe the recommendation is below 30GB. A shard is equate to a file. A file of 30GB is kind of big already. It makes moving and recovery longer. I have not experienced any data missing due to rebalancing yet. If your concern is theoretical, then I would probably not worry about it.