So I decided to put more nodes per physical host. These are not VMs it's 3 physical instances running on the same physical host.
For each host, I have 4 total servers....
I disabled shard allocation downed the host, rebuilt it with 3 instances and brought up the 3 instances, re-enabled shard allocation, rinse and repeat.
It seems some hosts got primary and replicas...
Is the cluster.routing.allocation.same_shard.host: true enough? I'm pretty sure I have it setup for all instances!
The new index on the far right seems ok.