Not really a good reason to be honest. Just historical legacy / cruft. It's not technically a cluster-level setting based on how it works, so you can't set it dynamically via /_cluster/settings. It's technically an index-level setting... but you can only update index-level settings through the API on an actual, existing index. Which is why you cannot set it before the index is created.
So it occupies this bizzaro role of being an index-level-but-also-cluster-level setting, which happens to only be configurable via the static config.
It really was just a hacky thing put in place a loooong time ago, which didn't follow the normal semantics of how settings should work. Which is why it's a bit of a special snowflake, and also why it's been removed 