Unbalanced distribution of index (shards) between HOT servers ES

Hi community !

I apologize if this topic has been covered for a long time.
But I never found it in the open Elastic Сommunity. :face_with_peeking_eye:

We have 11 cluster servers in the infrastructure ES:
Servers HOT - 3
Servers WARM - 3
Servers COULD - 2
Servers Master - 3

Some time ago I noticed that on HOT servers there is a discrepancy in disk space utilization.
The HOT 1 server began to fill up disk space faster than on the HOT 2 and HOT 3 servers.

I don’t understand why there is an uneven distribution of indexes (shards) between HOT servers in the cluster.

Sending to these three servers HOT-1 HOT-2 HOT-3 is configured on all logstash servers in the output section.

Can anyone suggest what are the special parameters for properly balancing the distribution of indexes ?

How to understand this and, most importantly, how to set up the correct distribution of indexes on servers with the HOT role ? :astonished:

Moreover, on all other servers with the roles of WARM and COULD, the distribution of the index record is even.

How have you setup the allocation filtering you are using?

Sending to three HOT servers is configured on all logstah in otptut.

Or are you talking about some other allocation ? :thinking:

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

I figured out the reason, the topic is closed.

It'd be great if you could share your solution, it might help someone in future.

At first, on the first and then on the second HOT servers, I had more than 85% full.
New shards could not be recognized (unsigned).
found ~ 1000 unsigned shards.

After I corrected the ILM policy by expanding storage towards WARM servers, all shards recovered, in less than a week the space on XOT servers stabilized.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.