Shard reallocation and disk space

Elastic Stack Elasticsearch
1

Hello, I'm using ElasticStack 7.6.1 on CentOS 8.

I recently upgraded the kernels on all three of our nodes so I then rebooted them all; after they were rebooted the cluster decided to reallocate some shards but for some reason it placed many large shards on one node to the extent that the node ran out of disk space and the cluster became inoperable. I was able to fix it by simply adding more space to the node and then rebooting all nodes, the cluster then evenly balanced the shards and roughly the disk space on each node.

I have a couple of questions regarding this though:

  1. Shouldn't the disk high and low water mark settings have stopped the disk on the node from becoming full i.e. it should've stopped trying to move shards to that node once its watermark settings had been reached? (I haven't changed the settings so they are at the default).

  2. As the disk did become full, If I didn't have the option of adding more space to the node then what else could I have done to force it to move data off that node? As far as I could tell the cluster was inoperable, so I'm not sure how I would've fixed this issue without first adding more space.

Thanks for any help.

2

Was your data path it's own filesystem? If it's shared, particularly with the FS containing /var, Elasticsearch data is in competition for other uses for disk space. Bad things result :slight_smile:

In that case, freeing space, say in /var/log, will let Elasticsearch resume.

3

Hi rugenl thanks for replying, no they're not shared; all 3 nodes had a 5.5 TB (now expanded to 8 TB) dedicated XFS file system on a separate disk for Elasticsearch data.

If I recall correctly each node had about 2 TB of free space during the reallocation and I'd also stopped log ingestion whilst it was moving shards about, so I'm perplexed at how it managed to completely fill one node's disk.

4

Can you tell if it was one large index or a lot of indices? It could be an index with 1 primary and 0 replica, that would live on only one node.

5

Hi, unfortunately not since the cluster has now rebalanced itself successfully so I don't know precisely which index shards had moved to the problematic node. We don't have replicas due to not having the disk space to keep replicas as well as be able to retain the data for as long as we need to.

Our largest indexes are the ones for windows event logs, these indexes are composed of 33 shards which I assumed would be evenly balanced across 3 nodes.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.