### Author:  [@adityajindal](https://discuss.elastic.co/u/adityajindal)
#### Post date: [September 24, 2017, 10:37am UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/1 "2017-09-24T10:37:25Z")
Hi
We are currently having a very large cluster with about 100 TB data in it. We want our snapshot in Amazon S3 Bucket to complete as fast as possible.
With reference from this
[https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html)
&
[https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html)
I tried increasing the maximum number of threads in the snapshot thread pool. The number of active threads increased during the Snapshotting process without any increase in the Snapshotting speed.
Currently I am getting 200 MBps Speed of Snapshot/Restore the data onto S3 Bucket
I have checked out that my S3 Bucket has far more limits (Around 2 GBps) than this.
Also I when I am doing a restore of the a snapshot from a Amazon S3, I am able to see threads active from the Generic Thread pool, instead of being active from the Snapshot thread pool.
What could be wrong here?
Thanks in advance!
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [September 25, 2017, 7:27am UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/2 "2017-09-25T07:27:44Z")
There are some internal throttling which applies to all kind of repositories according to:
>
Basically you can try to change `max_snapshot_bytes_per_sec` on your S3 repository settings. Something like (untested):
```auto
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"bucket": "my_bucket_name",
"max_snapshot_bytes_per_sec": "100m"
}
}
```
This setting is the rate per node as explained in [https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html)
HTH
---
### Author:  [@adityajindal](https://discuss.elastic.co/u/adityajindal)
#### Post date: [September 25, 2017, 8:58am UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/3 "2017-09-25T08:58:01Z")
Hi @dadoonet,
Thanks for your reply.
I forgot to mention that for my repository settings had the parameters **max\_snapshot\_bytes\_per\_sec** & **max\_restore\_bytes\_per\_sec** as **10gb** each.
For further reference my settings are
```
"my_repo" : {
"type" : "s3",
"settings" : {
"bucket" : "bucket_name",
"chunk_size" : "1gb",
"server_side_encryption" : "false",
"max_restore_bytes_per_sec" : "10gb",
"buffer_size" : "100mb",
"base_path" : "/snapshots/",
"region" : "us-east-1",
"max_snapshot_bytes_per_sec" : "10gb"
}
},
```
From the same documentation I have tried decreasing the buffer\_size so that we can have multiple multipart upload to S3. Couldn't get any significant improvements.
While browsing the code I came across [this](https://github.com/elastic/elasticsearch/blob/cd2a4372b49aa595b1ad811d00a90a0ffd3e81d2/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Repository.java#L47) setting for increasing the number of Concurrent Streams which write to S3.
How can I tweak this setting?
Thanks in advance!
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [September 25, 2017, 9:12am UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/4 "2017-09-25T09:12:04Z")
Sounds like a copy and paste from other repositories javadoc. 🙂
I don't know but @Igor_Motov might know.
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [September 25, 2017, 3:24pm UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/5 "2017-09-25T15:24:04Z")
@dadoonet this setting is a leftover from the initial implementation of snapshot restore in 1.0 when each repository was managing its own thread instead of using the common thread pool. We need to clean this up.
@adityajindal is your cluster located in AWS east as well or it's located outside of the AWS? Could you run [snapshot status](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-snapshots.html#_snapshot_status) command two times with 1 min interval between the runs and share the results (you can PM them to me if you don't want to post them publicly).
---
### Author:  [@adityajindal](https://discuss.elastic.co/u/adityajindal)
#### Post date: [September 25, 2017, 6:27pm UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/6 "2017-09-25T18:27:22Z")
Hi @Igor_Motov,
My cluster is located in the same region as the s3 Bucket (us-east-1).
Please find the Output of snapshot status 1 min apart [here](https://pastebin.com/FEhNqBns) & [here](https://pastebin.com/bsPH95Fg).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [September 26, 2017, 1:24pm UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/7 "2017-09-26T13:24:16Z")
I can see that it tries to upload on several threads but at the end of the day it looks like it's throttled to about 75mb per second. Did you try to upload a large file to S3 from this machine directly? I wonder what kind of throughput you get if elasticsearch is not involved to rule out AWS network throttling. Where do you have your data stored? What type of instance is it?
By the way, @tlrx please correct me if I am wrong, but I don't think reducing `buffer_size` would increase performance since because of sequential nature of uploads. If anything it should probably decrease it because of additional request overhead.
---
### Author:  [@adityajindal](https://discuss.elastic.co/u/adityajindal)
#### Post date: [September 26, 2017, 1:37pm UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/8 "2017-09-26T13:37:34Z")
Hi
I have tried uploading a large file to S3 from this machine directly.
The throughput I am getting is around 550 MBytes Per second for a large file. I was able to open 350 Connections. Ref: [Here](https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/)
I am using i3.16xlarge instance type to store my data.
From [this](https://www.elastic.co/guide/en/elasticsearch/plugins/5.6/repository-s3-repository.html#repository-s3-repository) documentation buffer\_size is the largest size for Single Part Upload. Assuming Multipart Upload gives better throughput, shouldn't decreasing this setting increase the number of Multipart Uploads and hence better throughput ?
Thanks in Advance!
---
### Author:  [@tanguy](https://discuss.elastic.co/u/tanguy)
#### Post date: [September 26, 2017, 2:26pm UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/9 "2017-09-26T14:26:22Z")
> [@Igor\_Motov](#):
>
> By the way, @tlrx please correct me if I am wrong, but I don't think reducing buffer\_size would increase performance since because of sequential nature of uploads. If anything it should probably decrease it because of additional request overhead.
That's also what I think. To execute a snapshot on S3, the snapshot service lists the shard's files to save and compares the length of the file with the chunk\_size parameter (1Gb for S3 repositories by default). If the file is larger than 1Gb then it will be split into multiple chunks. If the file is lower than 1Gb there will be 1 chunk. Then the snapshot service starts to bufferize the file and if the buffer is full it sends the file in multiple uploads (2 requests + 1 request per full buffer). If the buffer is not full the file is sent using a single request.
A too low buffer\_size can increase the number of requests, to the point where S3 limits are exceeded. But a low buffer\_size is helpful on unstable networks because only small parts of file will have to be re sent if the network dropped or the request failed. The best advise is to experiment because it depends of the network and the indices. Having the same values for chunk\_size and buffer\_size can also help to find the right settings.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2017, 2:26pm UTC](https://discuss.elastic.co/t/increase-snapshotting-speed-while-taking-a-snapshot-in-amazon-s3-bucket/101597/10 "2017-10-24T14:26:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.