All primary shards of some indices keep getting assigned to 1 of 14 data nodes

[danksim (Daniel Sim)]

ES: 7.6.x
3 coordinator, 3 master, 14 data
6 shards, 1 replica

All primary shards of some indices keep getting assigned to only 1 of 14 data nodes. So data node 12 has 6 primary shards of index x, y, z, etc. The other 13 nodes are normal.

This results in high JVM heap usage for data node 12 and also unassigned shards (reason: circuitBreakingException).

Why would this happen for only data node 12?

What I have tried:

1. Set -XX:CMSInitiatingOccupancyFraction=50
2. Find each index with all of its primary shards on data node 12 and set index.routing.allocation.total_shards_per_node to 1 or 2
   a. This helps but all primary shards of some new indices still get allocated to data node 12

What I have not tried:

1. Set cluster.routing.allocation.total_shards_per_node in index template
   a. Not sure how this would work when setting index.routing.allocation.require._name (I cannot use ILM for some reason so I wrote the shrink process myself and I need all primary shards for an index on the same node in order to shrink. See here)
2. Change circuitBreaker settings

EDIT: a coworker found this and it looks like this is the problem.

[coudenysj (Jachim Coudenys)]

I keep having the same issue.

All primary shards of new incides are assigned to one (almost always the same hot node) node.

[danksim (Daniel Sim)]

We solved this problem by scaling up our JVM heap size from 8g to 12g (i.e. -Xmx12g -Xms12g) per data node (and kubernetes statefulset memory from 16Gi to 24Gi).

Once this change was applied, we waited until the problematic node's shard count went up to par with the other nodes and then the cluster intelligently rebalanced itself by relocating the primary shards of the newly created indices off of the problematic node.