Increase Snapshotting Speed while taking a snapshot in Amazon S3 Bucket

dadoonet · September 25, 2017, 7:27am

There are some internal throttling which applies to all kind of repositories according to:

elastic/elasticsearch/blob/350125ed2a230a418d5a8ce517d2f48dd2e0cb83/core/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java#L236


/**
 * Constructs new BlobStoreRepository
 *
 * @param metadata       The metadata for this repository including name and settings
 * @param globalSettings Settings for the node this repository object is created on
 */
protected BlobStoreRepository(RepositoryMetaData metadata, Settings globalSettings, NamedXContentRegistry namedXContentRegistry) {
    super(globalSettings);
    this.metadata = metadata;
    this.namedXContentRegistry = namedXContentRegistry;
    snapshotRateLimiter = getRateLimiter(metadata.settings(), "max_snapshot_bytes_per_sec", new ByteSizeValue(40, ByteSizeUnit.MB));
    restoreRateLimiter = getRateLimiter(metadata.settings(), "max_restore_bytes_per_sec", new ByteSizeValue(40, ByteSizeUnit.MB));
    readOnly = metadata.settings().getAsBoolean("readonly", false);


    indexShardSnapshotFormat = new ChecksumBlobStoreFormat<>(SNAPSHOT_CODEC, SNAPSHOT_NAME_FORMAT,
        BlobStoreIndexShardSnapshot::fromXContent, namedXContentRegistry, isCompress());
    indexShardSnapshotsFormat = new ChecksumBlobStoreFormat<>(SNAPSHOT_INDEX_CODEC, SNAPSHOT_INDEX_NAME_FORMAT,
        BlobStoreIndexShardSnapshots::fromXContent, namedXContentRegistry, isCompress());
}


@Override

Basically you can try to change max_snapshot_bytes_per_sec on your S3 repository settings. Something like (untested):

PUT _snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "my_bucket_name",
    "max_snapshot_bytes_per_sec": "100m"
  }
}

This setting is the rate per node as explained in https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

HTH