I’m running Elastic v8.8.2 on Kubernetes and sometimes, snapshots only partially succeed with error message AmazonS3Exception\[Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown;. Some shards are failing due to this error - these shard files have a shared s3 prefix of <elastic_cluster_name>/<k8s_cluster_name>/indices/<index_hash>.
I'm using the default settings so a upload speed of 40mb/s (so already not that fast) and 3 retries with exponential backoff (defaults of the s3 module). Looking for advice on things I can try to address the error other than reducing the upload speed or increasing the number of retries.