# How to calculate no. of request going to s3 while snapshot **URL:** \ **Category:** Elasticsearch\ **Created:** [June 21, 2019, 3:21am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789 "2019-06-21T03:21:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Sukanta\_Saha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukanta_saha/32/117858_2.png) [@Sukanta\_Saha](https://discuss.elastic.co/u/Sukanta_Saha)\ **Post date:** [June 21, 2019, 3:21am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789/1 "2019-06-21T03:21:45Z")
Hi We are running an elasticsearch cluster with 40 data nodes and 3 master nodes data nodes are i3.4xlarge and master nodes are m5.xlarge, using es version 5.4.0 We are taking daily backup on s3 using snapshot command , here is the repository settings ``` "type": "s3", "settings": { "bucket" : "bucket", "chunk_size" : "100mb", "server_side_encryption" : "true", "max_restore_bytes_per_sec" : "10gb", "use_throttle_retries" : "true", "max_retries" : "20", "compress" : "true", "base_path" : "AA-cluster" } ``` While taking snapshot getting this error IndexShardSnapshotFailedException[Failed to write file list]; nested: IOException[com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 200; Error Code: SlowDown; Request ID: 9F00739FB2DEEAD2), S3 Extended Request ID: zFhvAAxsfAKBmkTWtRb+qz/bs3/wGQ5H7Vduc8rwRhWeOAvPApZVkZpvOvD3bt78y7m8WB/4eOs=]; nested: AmazonS3Exception[Please reduce your request rate. (Service: Amazon S3; Status Code: 200; Error Code: SlowDown; Request ID: 9F00739FB2DEEAD2)]; ", We checked with aws they wanted to how many request we are making to s3 The request rate is 3,500 requests per second, they want to know whether we are crossing this limit Any one has any idea how to calculate the no of request we are making while snapshot The cluster size is 93tb Thanks Sukanta ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 21, 2019, 6:46am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789/2 "2019-06-21T06:46:35Z")
> [@Sukanta\_Saha](#): > > Any one has any idea how to calculate the no of request we are making while snapshot The number of requests is largely determined by the number of [segments](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-segments.html) that have changed since the last snapshot, and the number of shards that are being snapshotted. Each segment to be uploaded is split into chunks and each chunk corresponds with at least one request. I notice that you've reduced the `chunk_size` from the default of `1gb` down to `100mb`, which will increase the number of requests on larger segments by a factor of 10. I would recommend changing this back to the default. You don't say how many shards you have in your cluster, but as a general rule fewer, larger, shards will take fewer requests to snapshot. Elasticsearch does not currently limit its request rate to S3, but that sounds like a feature that might be possible to implement. Please open a feature request on [Github](https://github.com/elastic/elasticsearch/issues) if you think this would be useful. ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 21, 2019, 9:06am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789/3 "2019-06-21T09:06:09Z")
Hi, To hopefully add to David's suggestion: How many snapshots are in that repo? How many objects are in the bucket/prefix (the one used by this repo)? How many indices/shards are in that cluster? Do you force merge to N before doing snapshots, do you use force merge at all? It would be interesting to see if you would get the same results in a newer version too with the version bump to the AWS SDK and underlying code making the S3 repository work. I believe much work went into this since 5.4 and the aws SDK also went up. You don't have a dev cluster with 93TB of data by any luck? 5.4.0 is old though, do you have a plan to move up to 5.6 than 6.8? Did you try setting a huge(maybe huge huge) number in max\_retries? Maybe 20 is simply not enough to successfully build the file(blob) list with a such a huge snapshot in the same repo(bucket). You could also try without throttled retries, did you? But yeah, based on my current knowledge of S3 app architecture, this should be considered an application side bug or mis configuration, in the sense that you should be able to manage PBs in a single S3 bucket with billions of objects and the client code should always be able to retry until it gets done if the errors are transient and not fatal; throttling, 5XX, back pressure, etc. Even if you reduce your object count by 10 or any other factor, someone else _could_ have a snapshot that's simply bigger by that same factor which would bring him back to the same level. It's not an S3 problem to have billions of small objects in a bucket/prefix, it's just cost more money than if you'd store the same in less objects. It's use case dependent. If the s3 repository plugin doesn't retry until it succeed with correct exponential backoff policies when doing atomic operations then you're in a pickle or will be at some point in the future. Even more if you can't set or control further the AWS SDK S3 client that is used under the hood to customize its retry/throttling behavior for your big use case. Looking at the code this exception would _appear_(not an expert on that) to come from code that build a representation of what is in the bucket and not the part that actually puts the chunks/blobs which to me make sense since 3500req/s X 100MB/req = 340GB/s. Not saying you can't theoretically push that in S3 and make 3500 PUT of 100MB per second but you definitively can't do that with 40 i3.4xlarge capped at 1.25GB/s of network throughput... IF you are being throttled while PUTing the chunks it's not at 3500req/s of 100MB each for SURE. But I believe S3 can return this at much lower rate while it adjust the S3 backend to scale up to your required request rate for a prefix. Which would also point to the application side not tolerating the throttling correctly and fatally giving up instead of adjusting. Profiling the request rate would indeed shed light on this part. Maybe it's Listing/GETing/HEADing to sync it's local representation of the bucket and that's when the throttling triggers enough retries to make the plugin abandon. Maybe @DavidTurner knows for sure if part of the operation actually consist in building/syncing such a local representation of the bucket content, most S3 apps need to do that but I'm not familiar enough with the s3-repository logic to know for sure. For your specific question to help in your investigation of the rate produced by the plugin: Metrics approach: 1. [How Do I Configure Request Metrics for an S3 Bucket?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics.html) 2. If you have many things in that same bucket which are unrelated. [How Do I Configure a Request Metrics Filter?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics-filter.html) 3. [Concept and metric description](https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html) 4. [Concept continued and config](https://docs.aws.amazon.com/AmazonS3/latest/dev/metrics-configurations.html) Logs approach (then count/aggregate/analyse those logs to derive requests rate): 1. [Server Access Logging delivered to S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html) 2. [Cloudtrail (with the S3 data event logging enabled)](https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudtrail-logging.html) So for cloudtrail it would mean enabling S3 data events than sending cloudtrail to an S3 and then either downloading this for analysis or stuffing it into something like ES or querying it with Athena or something else to derive the metrics you're interested in. Could also send cloudtrail to cloudwatch logs to derive metrics from logs via either [cloudwatch logs metric filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringPolicyExamples.html) or [cloudwatch logs insight](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html). For server access logging it means enabling them for the bucket and then downloading them/stuffing them in a tool for analysis like ES or Athena or something else. I presented the options in the order I recommend you try them (from easiest to hardest). I would also recommend you ask AWS support how you should obtain the numbers they are asking for... Maybe they get can it from their backend without you doing anything... or maybe they have a suggestion I didn't mention/know about? Other possibilities could involve routing all the request from the s3-repository plugin through a proxy like squid with access logs enabled and then doing the log analysis like above but on those logs... Maybe ES can be configured to log in debug in a way that this plugin will configure the AWS SDK it uses to log every requests it makes to S3 in an ES log file on each nodes. I don't know if that's possible or how to do it if it is. Require someone else to enlighten you on that. Then, gather and analyse like above. ---
**Author:** ![Sukanta\_Saha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukanta_saha/32/117858_2.png) [@Sukanta\_Saha](https://discuss.elastic.co/u/Sukanta_Saha)\ **Post date:** [June 24, 2019, 8:10am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789/4 "2019-06-24T08:10:04Z")
> [@martinr\_ubi](#): > > How many snapshots are in that repo? > How many objects are in the bucket/prefix (the one used by this repo)? > How many indices/shards are in that cluster? > Do you force merge to N before doing snapshots, do you use force merge at all? Thanks David and Martinr for your help David I will try to use chunk\_size 10gb and see if it helps Martinr Here I tried to answer your question How many snapshots are in that repo? We keep last 3 days snapshots , remaining we delete How many objects are in the bucket/prefix (the one used by this repo)? This bucket has 256 hexadecimal partitions on it, for example spr-prod-es-backup-group4/[0-9A-F][0-9A-F] spr-prod-es-backup-group4-dr/[0-9A-F][0-9A-F] So far we have 21 partitions created like this 0B 0C 0D 0E 0F 1B 1C 1D 1E 1A 0A 6A 2A 3A 4A 5A 7A 1F 8A 9A AA How many indices/shards are in that cluster? No. Of Indices : 551 active\_primary\_shards : 4401 Doc Count : 122849007783 Do you force merge to N before doing snapshots, do you use force merge at all? No we do not force merge And we are using repository-s3-5.4.3-SNAPSHOT.zip ---
**Author:** ![Sukanta\_Saha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukanta_saha/32/117858_2.png) [@Sukanta\_Saha](https://discuss.elastic.co/u/Sukanta_Saha)\ **Post date:** [June 27, 2019, 1:26am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789/5 "2019-06-27T01:26:17Z")
By making chunk size 1gb it improves a lot Now the settings is follows { "spr-prod-es-backup-group4" : { "type" : "s3", "settings" : { "bucket" : "spr-prod-es-backup-group4", "chunk\_size" : "1gb", "server\_side\_encryption" : "true", "max\_restore\_bytes\_per\_sec" : "10gb", "use\_throttle\_retries" : "false", "max\_retries" : "100", "compress" : "true", "buffer\_size" : "1gb", "base\_path" : "AA-listening-tw-firehose4" } } } We kept chunk size and buffer size same because in doc it said When the chunk size is greater than the value of the buffer\_size, Elasticsearch will split it into buffer\_size fragments and use the AWS multipart API to send it Can you suggest is this settings is correct Also now we are seeing a different error while snapshoting "state" : "PARTIAL", "start\_time" : "2019-06-26T04:15:20.882Z", "start\_time\_in\_millis" : 1561522520882, "end\_time" : "2019-06-26T04:30:39.165Z", "end\_time\_in\_millis" : 1561523439165, "duration\_in\_millis" : 918283, "failures" : [ { "index" : "tweet\_p999999\_v\_2\_20190430\_1345", "index\_uuid" : "tweet\_p999999\_v\_2\_20190430\_1345", "shard\_id" : 4, "reason" : "IndexShardSnapshotFailedException[Failed to write file list]; nested: NoSuchFileException[Blob [pending-index-112] does not exist]; ", "node\_id" : "R7uaWsplQNmrOiqCEHQD4w", "status" : "INTERNAL\_SERVER\_ERROR" } ] Any one has any idea why this error is coming ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2019, 1:26am UTC](https://discuss.elastic.co/t/how-to-calculate-no-of-request-going-to-s3-while-snapshot/186789/6 "2019-07-25T01:26:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.