Disk space issues on 7.10

Elastic Stack Elasticsearch
1

I have two more or less identical servers in my cluster that each have 3 disks in the data_path each with 1.5TB.
Recently one of the server has started running out of disk on one of the three disks (the first in the list if it matters).

Over all there is plenty of disk space so the high water mark is never reached ES just suddenly runs out of space on one disk then and shuts down.

2

At a guess, that might come down to the differing shard sizes on the paths.

3

Thanks Mark, how would I check?

4

Is shard allocation equal on all nodes?

5

This seems weird for a couple of reasons:

  1. the disk watermarks are per-node things: if one node starts to run out of space then it will hit its watermark, even if there's plenty of space overall.

  2. when a node runs out of space it doesn't shut down.

Can you clarify what you mean exactly, with logs and so on? We only check disk space every 30s or so, so if you write a huge amount of data in 30s and/or the watermarks are set very close then it is possible to go from below the high watermark to a full disk before any protection mechanisms kick in.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.