Index size explodes after _split

Elastic Stack Elasticsearch
1

Recently I had to split an index that had grown to 100GB on a single shard. I used the Split API to split this into 8 shards which nearly instantly created the new index with all 8 shards on the same node. I deleted the old index and created an alias from the new one

I noticed the primary store size had grown from 100GB to about 650GB. When enabling replicas and waiting for the cluster to re-balance I ran into out of disk issues over the weekend, causing my entire cluster to go into read-only mode.

I deleted the replicas again which freed up some disk space and unblocked the cluster again.

Over the course of a week (!) the index shrunk back down to its normal 100GB size on 8 shards.

How can I prevent this from happening in the future? Is there a way to split an index without its size exploding? Can forcemerge be used for this and what will be the impact if the index is used constantly for reading and writing?

2

This is expected, per the docs;

Splitting works as follows:

  • First, it creates a new target index with the same definition as the source index, but with a larger number of primary shards.
  • Then it hard-links segments from the source index into the target index. (If the file system doesn’t support hard-linking, then all segments are copied into the new index, which is a much more time consuming process.)
  • Once the low level files are created all documents will be hashed again to delete documents that belong to a different shard.
  • Finally, it recovers the target index as though it were a closed index which had just been re-opened.
3

So @warkolm , would you recommend running a forcemerge first, before enabling replicas? Splitting by a significant factor can easily fill up the disks on a decent sized index.

4

I don't think that would help, doing one after should though.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.