# Snapshot performance with ES 7.10 **URL:** \ **Category:** Elasticsearch\ **Created:** [January 11, 2021, 3:27pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739 "2021-01-11T15:27:13Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [January 11, 2021, 3:27pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/1 "2021-01-11T15:27:13Z")
Hi all: We have upgraded to ES 7.10 from 6.8. After the upgrade, we are observing that the time taken for the Snapshot on 7.10 is around 3 times slower compared to the snapshots with 6.8. These are snapshots of the ES cluster to an NFS mounted share. There is no change in the configuration from 6.8 to 7.10. I was wondering if anyone has this behavior. If so, are there any settings that need to be configured in ES 7.10 to get back the previous performance? Thanks. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 11, 2021, 6:01pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/2 "2021-01-11T18:01:21Z")
Is it still slow if you move to a fresh repository? ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [January 11, 2021, 7:00pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/3 "2021-01-11T19:00:36Z")
Hi Dave, The slowness is happening on a fresh repository as well with a new (greenfield) installation of ES 7.10. Thanks, -Karun ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 5:35am UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/4 "2021-01-12T05:35:34Z")
Ok, thanks for checking that. I'm not aware of any performance regression in this area. Can you quantify it in more absolute terms? How large/complex a snapshot are you taking (data volume, number of shards, any other pertinent info) and how long does it take? ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [January 12, 2021, 5:56pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/5 "2021-01-12T17:56:19Z")
Here are the details that you are asking for: ```auto =6.8 ES= curl get xxxxx:9200/_cat/snapshots/SnapshotRepo_1?v&s=id id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards eca95dae-1774-40be-921e-9311e5206263 SUCCESS 1609997090 05:24:50 1609997354 05:29:14 4.3m 3 12 0 12 =7.10 ES= curl get xxxx:9200/_cat/snapshots/SnapshotRepo_2?v&s=id id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards eddfcb8c-02ab-4dc6-94e9-ab654a60ba51 SUCCESS 1610002940 07:02:20 1610003581 07:13:01 10.6m 3 12 0 12 ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 5:59pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/6 "2021-01-12T17:59:21Z")
How large were these indices? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 6:03pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/7 "2021-01-12T18:03:22Z")
Also note that snapshots will re-use previously-snapshotted data where possible, so if the repository wasn't empty then that will confound your measurements too. ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [January 12, 2021, 6:13pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/8 "2021-01-12T18:13:56Z")
The repository was empty as we trying to compare performance of 6.8 vs 71.0. Also, there are 3 indices and only one of them is big, maybe around 6.2.GB. The other ones are in few MB or even smaller. Thanks, Karun ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 7:07pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/9 "2021-01-12T19:07:19Z")
Thanks. Yes, 10 minutes does seem longer than expected to make a ~6GB snapshot. The only relevant setting I can think of is `max_snapshot_bytes_per_sec` which defaults to `40mb`, i.e. 40MB/s, but of course that applies to both versions. There have been quite a lot of changes to how snapshots work between 6.8 and 7.10 but (as I said) I'm not aware of any that would cause such a performance drop. Unless anyone else has better ideas I think you'll need to share some logs from every node, with these settings: ```auto logger.org.elasticsearch.repositories: TRACE logger.org.elasticsearch.snapshots: TRACE logger.org.elasticsearch.cluster.service.MasterService: DEBUG ``` That will show a lot more detail on when things are happening and how long everything's taking. ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [January 15, 2021, 7:05pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/10 "2021-01-15T19:05:13Z")
Hi Dave, sorry for the late response. We will try with the settings you suggested and provide you the logs. ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [February 10, 2021, 5:42pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/11 "2021-02-10T17:42:49Z")
Hi Dave, we have the logs. Is there a way to upload them somewhere or let me know what look for? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 11, 2021, 10:37am UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/12 "2021-02-11T10:37:11Z")
Sure, I sent you a private message with an upload link. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 11, 2021, 11:03am UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/13 "2021-02-11T11:03:49Z")
I don't think you correctly applied the logger config I mentioned above. The logs only contain this: ```auto [2021-02-03T14:53:54,566][INFO][o.e.r.RepositoriesService] [REDACTED] put repository [REDACTED] [2021-02-03T14:53:54,803][INFO][o.e.s.SnapshotsService] [REDACTED] snapshot [REDACTED] started [2021-02-03T14:57:42,611][INFO][o.e.s.SnapshotsService] [REDACTED] snapshot [REDACTED] completed with state [SUCCESS] ``` There should be a good deal of additional tracing statements in between these two lines. Also, what exactly is the repository you're using? Is it a S3 bucket or is it something supposedly "S3-compatible"? Also also, the timestamps indicate that this snapshot took a little under 4 minutes to complete, much less than the 10 minutes you originally reported. Does this mean the problem is fixed? ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [February 11, 2021, 11:35am UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/14 "2021-02-11T11:35:12Z")
I believe we have applied the setting correctly. We are using the Data Domain as the respository. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 11, 2021, 12:43pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/15 "2021-02-11T12:43:33Z")
> [@karunelas](#): > > I believe we have applied the setting correctly. In which case there's something wrong with your logging setup, since the logs you shared contain no trace information at all. > [@karunelas](#): > > We are using the Data Domain as the respository As in the Dell EMC product? If so, we've definitely encountered performance issues with them in the past. Elasticsearch's access pattern did change between 6.8 and 7.10 and the new access pattern may not be very well-supported in the repository you're using. I would recommend running tests against real S3 to rule out the nonstandard repository as the source of problems first. ---
**Author:** ![karunelas](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@karunelas](https://discuss.elastic.co/u/karunelas)\ **Post date:** [February 11, 2021, 1:00pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/16 "2021-02-11T13:00:56Z")
I will recheck the log settings to ensure that they are correct. Yes, this is a Dell EMC product and we are from the same division as the Data Domain product. This is an enterprise product and I made changes to write it to S3. Where can I learn about the changes to Elastic's access pattern from 6.8 to 7.10. Appreciate any information on that. Thank you. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 11, 2021, 1:34pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/17 "2021-02-11T13:34:35Z")
You will be able to see the access pattern in the trace logs, and I believe you can log each access on the Data Domain side too. The details of the changes aren't really documented anywhere public, they're very much an implementation detail, but the main change I'm aware of is the heavier use of concurrent uploads. This gives a big performance boost on proper S3 buckets but I could believe that some third-party repositories might perform worse under a more concurrent workload. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2021, 1:35pm UTC](https://discuss.elastic.co/t/snapshot-performance-with-es-7-10/260739/18 "2021-03-11T13:35:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.