Nodes Space balancing Issue

Hi Team,

I am having cluster which has three data nodes, which are having a single master. We found that due to some hardware failure, one of the data noes got stuck and shut down. We restarted that nodes and it joined back successfully.

But while checking shard allocation, I have found that that node was having 90% of space utilised even though it has low number of shards compared to other two.

As node 1 and 3 are showing 200GB spared for each and node 2 was shown less than 80GB. Even though node 1 and 3 are showing more allocated shards compared to node 2.

Please help me how to solve this situation.

Shubham

Do you mean only a single master eligible node out of all 3?

No, Master node has been different node. I am having three data nodes and one separate master.

Ok, but are the data nodes master eligible?

One dedicated master node.
the others are data only and master eligible.

My one of the data node got down and problem starts. Now, that node allocated more space by elasticsearch and the other two nodes are healthy. I got certain watermarks.

Strange thing is that, the node which has space issues, has low number of shards.

What is the output from _cat/allocation?v.

shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
266 731.2gb 775.4gb 62.3gb 837.7gb 92 x.x.x.x x.x.x.x enode2
291 562.2gb 604.5gb 233.2gb 837.7gb 72 x.x.x.x x.x.x.x enode3
291 562.2gb 605.8gb 231.9gb 837.7gb 72 x.x.x.x x.x.x.x enode1

Hey man, any help here ?