Restore slow down as it progresses through more and more shards

Elastic Stack Elasticsearch
1

I have a elasticsearch cluster I am restoring to in AWS on ec2 instances.

Over the first 30 minutes or so of the restoration, the data throughput was over 500mbps but as the recovery was finishing through more and more shards it started slowing down. Near the end (last 10% of shards) It has slowed to under 50mbps. I noticed this when I was snapshotting as well.

I would like it to progress at the same pace throughout. Is there any reason why it slows down as it nears the end?

2

Where were you measuring the data throughput? On a per-node basis? On a global level?

Note that snapshot/restore throttles the restore rate on a per node basis (default 40mb per second, seting max_restore_bytes_per_sec).