We have 15 Tb data for approx. 100 indexes (with multiple shards and 1 replica). We have 6 data nodes. The problem we are facing is that sometimes a single node has all primary shards which increase the IOPS load on the machine. Is there way to rebalance primary shards and then replica shards for IOPS balancing.
It's unexpected that a primary should differ from a replica in this way since they should both perform roughly the same I/O. Can you share some more quantitative details, including the numerical measurements you have made?
The main exception to this rule is for an update-heavy workload, in which the primary must compute the updates. Are you using updates heavily? Your other recent question suggests that you use daily indices, which normally means your workload involves a lot of inserts but not many updates, but could you clarify?
Elasticsearch does not balance primary shards because they're mostly equivalent to replicas. However it's also a bit surprising that all the primaries end up on one node in a healthy 6-node cluster, particularly if you are using 1-replica redundancy, because with equal numbers of primaries and replicas the cluster would be very unbalanced if half the shards (i.e. all the primaries) were on a single node. Could you give some more details about this as it might be some kind of misconfiguration?
1 Like
Yes, we do not have updates, as our ELK is primarily used for logs ingestion. We have 100+ microservices sending their logs on their own indexes. We get approx 1TB data(with replica) over cluster set up of 3 -MI, 7 - DI with shards counts ranging from 7 to 14 for various indices. We data retention period of 15 days. As we use AWS instance, so we constantly get alerts where IOPS has crossed 50%.
So we were assuming that it was because of some data nodes having only primary shards of indices.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.