# \[Solved\] Improving Snapshot Recovery Speed

**URL:** <https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547>\
**Category:** Elasticsearch\
**Created:** [September 18, 2015, 7:40am UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547 "2015-09-18T07:40:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nathan\_F](https://avatars.discourse-cdn.com/v4/letter/n/b5a626/32.png) [@Nathan\_F](https://discuss.elastic.co/u/Nathan_F)\
**Post date:** [September 18, 2015, 7:40am UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547/1 "2015-09-18T07:40:32Z")

</div>

Hi all,

I have a cluster that I am working with on AWS that is 18TB is size and growing daily. Right now I create daily indices which are backed up to s3. I am looking at consolidating the data from several servers into just a few larger ones and was wondering if others have similar experiences.

During the startup of an entirely new cluster, I am making the following changes (taken from the log):

```
updating [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] to [50]
updating [indices.recovery.max_bytes_per_sec] from [40mb] to [2gb]
updating [indices.recovery.concurrent_streams] from [3] to [40]

```

From the \_snapshot api:

```
"daily_backup" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "my-bucket",
      "protocol" : "https",
      "base_path" : "daily",
      "max_restore_bytes_per_sec" : "4096mb",
      "max_snapshot_bytes_per_sec" : "200mb"
    }
  }

```

I realize that there are probably diminishing returns when setting larger numbers, but I am only seeing about 1GB/min of restoration per machine from s3 backups. The data is being written to raid0 non-EBS drives. Am I possibly missing something that would help speed this up? The AWS servers that I am using to test this must be able to pull s3 data more quickly than that (d2.2xlarge). I will take a look at setting max\_num\_segments = 1 (and redoing all of my backups) with the hope that this might help overall performance for restoration as well as daily function. Otherwise I would love to hear suggestions. If more information would be helpful, I am happy to oblige.

Note: I made a few changes to snapshot restoration api that allow me to trigger multiple simultaneous snapshot restorations at once. [https://github.com/elastic/elasticsearch/pull/12258](https://github.com/elastic/elasticsearch/pull/12258) (I never touch java so please don't judge what did there too harshly.)

---

<div class="post-metadata">

**Author:** ![Nathan\_F](https://avatars.discourse-cdn.com/v4/letter/n/b5a626/32.png) [@Nathan\_F](https://discuss.elastic.co/u/Nathan_F)\
**Post date:** [October 6, 2015, 8:33am UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547/2 "2015-10-06T08:33:22Z")

</div>

It is always the simpler things isn't it? I have my ES cluster behind a nat in a private subnet on amazon. Changing the nat's instance type to one that supports "high" network performance has quadrupled the speed at least. It looks like that is the only bottleneck.

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [October 6, 2015, 8:53am UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547/3 "2015-10-06T08:53:50Z")

</div>

Maybe try removing restore throttling altogether on the restore (set max\_restore\_bytes\_per\_sec to 0)?

This recent issue [https://github.com/elastic/elasticsearch/pull/13828](https://github.com/elastic/elasticsearch/pull/13828) means that ES is throttling much more than you requested.

If you do see a speedup, please report back!

---

<div class="post-metadata">

**Author:** ![Nathan\_F](https://avatars.discourse-cdn.com/v4/letter/n/b5a626/32.png) [@Nathan\_F](https://discuss.elastic.co/u/Nathan_F)\
**Post date:** [October 7, 2015, 4:02am UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547/4 "2015-10-07T04:02:34Z")

</div>

Does that setting interact with indices.recovery.max\_bytes\_per\_sec? Should I set both to zero?

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [October 7, 2015, 7:52am UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547/5 "2015-10-07T07:52:32Z")

</div>

I think recovery throttling is not affected by the above bug (only restoring a snapshot), so you shouldn't need to set indices.recovery.max\_bytes\_per\_sec to 0 (unless you separately want to!).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:46pm UTC](https://discuss.elastic.co/t/solved-improving-snapshot-recovery-speed/29547/6 "2017-07-05T23:46:16Z")

</div>