I have a job that creates a new version on an index every morning then deletes the old version of the index when it is done. About 3/7 days per week this job fails. What I am observing is that sometimes when trying to create the new index shards are either not being assigned at all or taking a long time to be assigned (around 10 minutes).
The index is small (~100k entries) and cluster setup is really simple. There are 4 data nodes in the cluster and the index is 1 shard with 3 replicas. So there should be one primary shard and one replica on each of the rest of the 3 remaining nodes in the cluster.
Any insight on why this is happening would be greatly appreciated. Thanks!