Hello,
We are running an ES 5.16 cluster in 3 DC's (total of 15 hosts, each host is 8 cpu / 32 GB ram and we have heap set to 16GB).
DC1 = zone1 (1 master node, 6 data nodes)
DC2 = zone2 (1 master node, 6 data nodes)
DC3 = zone1 (master only)
Reason for the DC3 with only a master is for redundancy in event we lose an entire DC (without having to enable multiple masters in each DC).
On all 15 nodes we have:
node.attr.rack: r1
Currently we have the awareness set to zone:
cluster.routing.allocation.awareness.attributes: zone
When we do maintenance work like OS patching or even bouncing, we do node by node to prevent any downtime. We will turn off shard allocation on the node we are working on so the shards that were on that node do not get reallocated. Then do the necessary work.
After doing the work, we then enable shard allocation once the node is up and joins the cluster.
Since our current setup will never have the Primary and Replica in the same DC (zone), it takes 30+ minutes for each shard in 1 index to sync up with the Primary.
The 1 index in question we have 15 primaries and 1 replica set. The size of each shard is approximately 8GB and growing daily.
We increased the max_bytes_per_sec to 200mb, and the concurrent_streams to 4.
While this did help on the overall issue, it didn't help any on the individual shard level (still takes 30+ minutes to recovery from primary due to cross DC copying.)
Is there anything we could do by changing awareness settings that could help out? My thought is that we will always have the potential for cross DC copying during recovery since the primary could be in the other DC regardless of how we setup the zones.
We are not using snapshots, but if this is a viable option to speed up recovery...we could look into that aspect.
Any ideas or suggestions would be appreciated.
