Apologies if this is a case of RTFM, but I read the Index Recovery Prioritization docs among others and couldn't find an answer.
I was descaling one of my Elasticsearch clusters, with a few hundred shard relocations running in parallel. Everything was dandy until a new hourly index was made. To my surprise, the new index's shard allocation was being throttled, because the full node_concurrent_incoming_recoveries quota was being exhausted by the shard relocations.
This has some pretty nasty consequences. If the new shards can't be allocated, then too much incoming data hits the few servers that do allocation, they start spiking their CPU and thrashing their heap, get unstable, disconnect, cause the cluster to go red... all kinds of fun!
Is there a way to always de-prioritize relocations, so that new index allocations are never throttled?