So I found out about the auto_expand_replicas setting from this thread while searching something else. I've set it to false and I will see if it works tomorrow when the new index is created.
EDIT: Marking this as the solution, since then the cluster configuration changed so I needed the replicas, as such I couldn't test it thoroughly, but this is as close of a solution I came, and I hope it can help someone.