Hello! i have a strange issue in my elasticsearch cluster
so i have 5 nodes ( 4 data and masters and 1 master only node )
so each node has 5.7 tb disk space on it
but on the first node my disk is almost completely full, and on the rest it is half full
the number of shards on all nodes is approximately the same
df -h from first node
/dev/mapper/vg1-data 5.8T 5.1T 717G 88% /var/lib/elasticsearch
and here is /cat/shards output
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
354 5tb 5tb 714.9gb 5.7tb 87 10.0.5.21 10.0.5.21 elastic-01
392 3.2tb 3.2tb 2.5tb 5.7tb 55 10.0.5.23 10.0.5.23 elastic-02
393 3.8tb 3.8tb 1.8tb 5.7tb 67 10.0.5.28 10.0.5.28 elastic-07
392 3.9tb 3.9tb 1.7tb 5.7tb 69 10.0.5.27 10.0.5.27 elastic-06
i tried summing the results from cat/shards | grep elastic-01 and it turned out that all shards on this node occupy 3.5 tb
but why does both elastic and du -hs show that more space is being used?
du -hs inside /var/lib/elasticsearch shows 5.1 tb too
du-hs*
5.1T nodes
4.0K range