Single node consuming more disk space

[purushothamkdr143 (Purushotham Reddy)]

Hi All,

I have setup Elasticsearch cluster using eck operator. details as follows.

master nodes: 3
data nodes: 9

only single data node is consuming more disk space although it has less number of shards than the other. Could you please let us know what could be reason for this? I know this is not sufficient to trouble shoot but on a high level what factors could be causing this?

[warkolm (Mark Walkom)]

What version are you on?
What is the output from _cat/allocation?v?

[purushothamkdr143 (Purushotham Reddy)]

Hi Mark Walkom,

I am using Elasticsearch version 6.8.14. Cluster settings & allocation as follows.

Allocation:

Cluster Settings

[warkolm (Mark Walkom)]

Please don't post pictures of text or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

[purushothamkdr143 (Purushotham Reddy)]

Sure.

here is the output of allocation(_cat/allocation?v).

shards disk.indices disk.used disk.avail disk.total disk.percent host ip
666 335.1gb 362.1gb 129gb 491.1gb 73 10.244.8.132 10.244.8.132
666 218.5gb 244.5gb 246.6gb 491.1gb 49 10.244.4.16 10.244.4.16
666 363gb 390gb 101.1gb 491.1gb 79 10.244.8.4 10.244.8.4
665 288gb 314.7gb 176.4gb 491.1gb 64 10.244.7.133 10.244.7.133
666 199.8gb 226.3gb 264.8gb 491.1gb 46 10.244.8.133 10.244.8.133
666 189.1gb 215.1gb 275.9gb 491.1gb 43 10.244.1.11 10.244.1.11
486 395gb 422.5gb 68.6gb 491.1gb 86 10.244.7.132 10.244.7.132
666 246.6gb 273.2gb 217.8gb 491.1gb 55 10.244.8.5 10.244.8.5
665 162.3gb 188.5gb 302.5gb 491.1gb 38 10.244.2.139 10.244.2.139

the nodes which has less number of shards i.e 486 is consuming more disk space(422.5gb) than others.

[warkolm (Mark Walkom)]

Ok but what's the average shard size on that node, it must be higher?

[purushothamkdr143 (Purushotham Reddy)]

Sorry, How can I check the average shard size?

I see there are some shards which have size more than 20GB.