My cluster has the usual 5 primary shards with 1 replica of each. Disks are running low on space and I'm looking at my options.
- Delete the oldest indices after doing a backup to cloud storage like AWS S3 or Glacier?
- Straight up delete old indices
- Attempt to shrink the indices down to 1 primary shard and 1 replica
Focusing on option 3 for a minute, the first step is to move all primary and replica shards for the single index onto the same node. Immediate implication of doing this will be that one node's disk space will be out of balance from the cluster. Other implication, with the defaults, you can only shrink down to 1 primary/replica shard if you started out with 5. Single point of failure, but better to have data a little longer than nothing at all?
Is the best practice to have a data node that's only used for this shrink purpose? Then maybe have a filter on that node, so that the only thing ever written to it are indices that are manually being shrunk?
How do you automate this when you have hundreds of indices that you want to re-allocate to shrink?
Any suggestions appreciated!