Shard reallocation and disk space

[kernelpanic (FreeBSD user)]

Hello, I'm using ElasticStack 7.6.1 on CentOS 8.

I recently upgraded the kernels on all three of our nodes so I then rebooted them all; after they were rebooted the cluster decided to reallocate some shards but for some reason it placed many large shards on one node to the extent that the node ran out of disk space and the cluster became inoperable. I was able to fix it by simply adding more space to the node and then rebooting all nodes, the cluster then evenly balanced the shards and roughly the disk space on each node.

I have a couple of questions regarding this though:

1. Shouldn't the disk high and low water mark settings have stopped the disk on the node from becoming full i.e. it should've stopped trying to move shards to that node once its watermark settings had been reached? (I haven't changed the settings so they are at the default).

2. As the disk did become full, If I didn't have the option of adding more space to the node then what else could I have done to force it to move data off that node? As far as I could tell the cluster was inoperable, so I'm not sure how I would've fixed this issue without first adding more space.

Thanks for any help.

[rugenl (Len Rugen)]

Was your data path it's own filesystem? If it's shared, particularly with the FS containing /var, Elasticsearch data is in competition for other uses for disk space. Bad things result

In that case, freeing space, say in /var/log, will let Elasticsearch resume.

[kernelpanic (FreeBSD user)]

Hi rugenl thanks for replying, no they're not shared; all 3 nodes had a 5.5 TB (now expanded to 8 TB) dedicated XFS file system on a separate disk for Elasticsearch data.

If I recall correctly each node had about 2 TB of free space during the reallocation and I'd also stopped log ingestion whilst it was moving shards about, so I'm perplexed at how it managed to completely fill one node's disk.

[rugenl (Len Rugen)]

Can you tell if it was one large index or a lot of indices? It could be an index with 1 primary and 0 replica, that would live on only one node.

[kernelpanic (FreeBSD user)]

Hi, unfortunately not since the cluster has now rebalanced itself successfully so I don't know precisely which index shards had moved to the problematic node. We don't have replicas due to not having the disk space to keep replicas as well as be able to retain the data for as long as we need to.

Our largest indexes are the ones for windows event logs, these indexes are composed of 33 shards which I assumed would be evenly balanced across 3 nodes.