Sharding hot vs warm Nodes

Nice summary. Here is a related discussion with several useful resources & guides. Suggest you start there.

Can you explain the "daily indices that are on the TB sizes sometimes and sometimes in the ranges above 20GB" - is that daily data of TB or what is the 20GB part?

More broadly, the guidance seems to be to hold shards to about 50GB, more or less, so your 83GB is on the large side and you probably should aim for less, like half.

Given your sizes, it seems this shard size limit will be your governing factor as long as your ingest & search speeds are acceptable. And you have enough hot/warm nodes that hopefully (not guaranteed) the shards will be spread around, improving performance. Also, simpler is better.

As far as I know there is no way to 'limit' the size - very large shards seem to perform poorly (single-threaded queries), and slow recoveries. However, you can use ILM to rollover the index when it hits a given size, so if your index has 20 shards and reaches 1TB in size, you're at 50GB per shard and can get a new index, etc.

Also watch your overall shard count, i.e. about 20 per GB of heap (maybe a little more in recent versions). People tend to miss this and end up with a thousand shards and stability problems. This & index count often drive people away from daily indexes.

For shrinking, especially for cold storage, I'd think you'd shrink down based on the 50GB target, so if you have a 250GB index with 10 shards, you could shrink to 5 shards for warm/cold storage.

Remember the ILM shrink operation moves all shards to a single node (a sad requirement of the shrink operation); for your 1TB+ indexes, this is a lot of data & movement, so watch cluster performance when this is going on.