Warm nodes on spinning disks not stable, falling out of cluster during recovery

It looks like you are pushing your nodes far harder than the defaults allow, and far harder than they can cope:

The default for this is 2 and increasing it is not recommended.

The default for this is 2 and increasing it is not recommended.

The default for this is 2 and increasing it is not recommended.

The default for this is 4 and increasing it is not recommended.

The default for this is 40mb; increasing it is ok if your nodes can cope with the increase, but the evidence indicates that 250mb is far too high for your warm nodes.

1 Like