Is it possible to configure replica shards to synchronize data?

[wangxr1985 (xinrong wang)]

By default, Elasticsearch replicates data from primary shards to replica shards. However, in some cases, this can lead to performance issues. For example:

1. When the cluster experiences high query volume and has many replicas (e.g., 1 primary shard and 100 replica shards), the node hosting the primary shard may experience higher CPU and bandwidth usage compared to other nodes.

2. When nodes in the cluster are distributed across multiple data centers, if the primary shard is in Data Center A and many replica shards are in Data Center B, all replica shards syncing data from nodes in Data Center A can consume the inter-data center dedicated bandwidth.

To address these issues, it would be beneficial to allow certain replica shards to sync data from other shards, thereby reducing the load on the primary shard. This can be achieved by implementing custom data synchronization strategies or utilizing advanced Elasticsearch features to optimize data replication across the cluster.

[DavidTurner]

@wangxr1985

When the cluster experiences high query volume and has many replicas (e.g., 1 primary shard and 100 replica shards), the node hosting the primary shard may experience higher CPU and bandwidth usage compared to other nodes.

Queries (i.e. searches) are spread across all shard copies equally, there's no extra load on the primary in this case.

@wangxr1985

all replica shards syncing data from nodes in Data Center A can consume the inter-data center dedicated bandwidth.

Cross-cluster replication is the best way to handle this.

[wangxr1985 (xinrong wang)]

Yes, query requests are evenly distributed across all shards. However, when there are a large number of write operations (especially update operations), the write load on the primary shard can become very high. With multiple replicas, the CPU and bandwidth usage of the node hosting the primary shard will be significantly higher than that of other nodes.

My current solution is to increase the number of primary shards from 1 to multiple, but this will increase the total number of shards, leading to a decrease in query performance and an increase in query load.

[DavidTurner]

@wangxr1985

Yes, query requests are evenly distributed across all shards. However, when there are a large number of write operations (especially update operations), the write load on the primary shard can become very high.

Ok, yes cross-cluster replication is the answer to this too.

[wangxr1985 (xinrong wang)]

Are you saying that multiple indexes can be created to synchronize index data using CCR?
For example, if there are 10 data nodes in data centers A, B, and C each, can we create three indexes, each set with 1 primary shard and 9 replica shards, and then ensure that the shards of these three indexes are distributed across the corresponding data centers, is that correct?
However, CCR is a paid feature. Are there any other ways to achieve this?

[DavidTurner]

@wangxr1985

Are you saying that multiple indexes can be created to synchronize index data using CCR?

Yes, you'd normally have one (or more) clusters in each data center, with CCR pulling the data from the central leader cluster. For really big setups you can use chained replication to further reduce load on the central cluster.

@wangxr1985

However, CCR is a paid feature. Are there any other ways to achieve this?

Not really. This kind of problem only arises when you're running a large cluster, which costs a lot of money anyway just for the infrastructure, so paid features like CCR actually end up saving more in infra savings than they cost.