On an ES 7.3 system: I have an index which gets created with 10 primary shards and 1 replica. I have an ILM policy which rolls over at 500GB, and then shrinks 10 days later to 5 shards and 0 replicas, expiring after 60 days (after rollover).
I don't know if this is optimal. I was thinking of changing it to 8 shards + 1 replica, rollover at 400 GB to 4 shards, same expiration. However, I am unsure what will happen when I change the policy. If I change the number of shards in the template and shrink shards to 8 and 4, when the next index rolls over, it will properly create the new index with 8 shards. That part is fine. But when the shrink step fires, it will try to shrink 10 shards to 4, which will error because it's not a multiple.
What's the best way to make these adjustments? My guess is to create a new policy instead of modifying the existing one. Then when I edit the template to change the policy, also change the number of shards. Close? Or is there more coordination required?
Followup question: Is there any purpose to not doing the shrink step until a few days after rollover? This is just for instance and container logs, nothing extravagant. Maybe just as good to do it a day or two later? Gives enough time for late logs to get ingested and indexed if the pipeline stalls for a bit.