Hello, I'm using ElasticStack 7.6.1 on CentOS 8.
I recently upgraded the kernels on all three of our nodes so I then rebooted them all; after they were rebooted the cluster decided to reallocate some shards but for some reason it placed many large shards on one node to the extent that the node ran out of disk space and the cluster became inoperable. I was able to fix it by simply adding more space to the node and then rebooting all nodes, the cluster then evenly balanced the shards and roughly the disk space on each node.
I have a couple of questions regarding this though:
Shouldn't the disk high and low water mark settings have stopped the disk on the node from becoming full i.e. it should've stopped trying to move shards to that node once its watermark settings had been reached? (I haven't changed the settings so they are at the default).
As the disk did become full, If I didn't have the option of adding more space to the node then what else could I have done to force it to move data off that node? As far as I could tell the cluster was inoperable, so I'm not sure how I would've fixed this issue without first adding more space.
Thanks for any help.