# Index size explodes after \_split

**URL:** <https://discuss.elastic.co/t/index-size-explodes-after-split/150692>\
**Category:** Elasticsearch\
**Created:** [October 2, 2018, 12:02pm UTC](https://discuss.elastic.co/t/index-size-explodes-after-split/150692 "2018-10-02T12:02:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdekker](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@cdekker](https://discuss.elastic.co/u/cdekker)\
**Post date:** [October 2, 2018, 12:02pm UTC](https://discuss.elastic.co/t/index-size-explodes-after-split/150692/1 "2018-10-02T12:02:59Z")

</div>

Recently I had to split an index that had grown to 100GB on a single shard. I used the Split API to split this into 8 shards which nearly instantly created the new index with all 8 shards on the same node. I deleted the old index and created an alias from the new one

I noticed the primary store size had grown from 100GB to about 650GB. When enabling replicas and waiting for the cluster to re-balance I ran into out of disk issues over the weekend, causing my entire cluster to go into read-only mode.

I deleted the replicas again which freed up some disk space and unblocked the cluster again.

Over the course of a week (!) the index shrunk back down to its normal 100GB size on 8 shards.

How can I prevent this from happening in the future? Is there a way to split an index without its size exploding? Can forcemerge be used for this and what will be the impact if the index is used constantly for reading and writing?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 4, 2018, 5:20am UTC](https://discuss.elastic.co/t/index-size-explodes-after-split/150692/2 "2018-10-04T05:20:44Z")

</div>

This is expected, [per the docs](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/indices-split-index.html#_how_does_splitting_work);

> Splitting works as follows:
> 
> - First, it creates a new target index with the same definition as the source index, but with a larger number of primary shards.
> - Then it hard-links segments from the source index into the target index. (If the file system doesn’t support hard-linking, then all segments are copied into the new index, which is a much more time consuming process.)
> - Once the low level files are created all documents will be `hashed` again to delete documents that belong to a different shard.
> - Finally, it recovers the target index as though it were a closed index which had just been re-opened.

---

<div class="post-metadata">

**Author:** ![cdekker](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@cdekker](https://discuss.elastic.co/u/cdekker)\
**Post date:** [October 17, 2018, 1:58pm UTC](https://discuss.elastic.co/t/index-size-explodes-after-split/150692/3 "2018-10-17T13:58:57Z")

</div>

So @warkolm , would you recommend running a forcemerge first, before enabling replicas? Splitting by a significant factor can easily fill up the disks on a decent sized index.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 17, 2018, 8:59pm UTC](https://discuss.elastic.co/t/index-size-explodes-after-split/150692/4 "2018-10-17T20:59:25Z")

</div>

I don't think that would help, doing one after should though.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2018, 8:59pm UTC](https://discuss.elastic.co/t/index-size-explodes-after-split/150692/5 "2018-11-14T20:59:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.