Creating Snapshot in Azure

code_blue · January 17, 2018, 1:04am

We have setup snapshot functionality using Azure Blob container. When we are creating a snapshot of a smaller Index and restoring it, we can see the new Index through Kibana.

However when we are trying to create a snapshot of an Index of size 30 GB, the query runs successfully but I am not able to see the backup in the Azure storage.
Does Elastic Search creates the snapshot of larger Indexes asynchronously or is there a limitation around this.

dadoonet · January 17, 2018, 7:41am

There is no limitation but there are some space for improvements for sure.
For example, we split the data into 64m blobs. (Azure repository plugin | Elasticsearch Plugins and Integrations [8.11] | Elastic)

This can be optimized like by fixing:

github.com/elastic/elasticsearch

Increase chunk_size max to 256 Mb instead of 64 Mb

opened 08:15AM - 02 Mar 17 UTC

closed 10:05AM - 31 Jul 18 UTC

dadoonet

>enhancement good first issue help wanted :Distributed/Snapshot/Restore

While working on an azure issue, I found [this documentation](https://docs.micro…soft.com/en-us/rest/api/storageservices/fileservices/understanding-block-blobs--append-blobs--and-page-blobs) which says that we can now increase the max block blob to 256mb instead of the default 64mb which has been introduced in #22577 > # About Block Blobs > Block blobs let you upload large blobs efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. You create or modify a block blob by writing a set of blocks and committing them by their block IDs. Each block can be a different size, up to a maximum of 100 MB (4 MB for requests using REST versions before 2016-05-31), and a block blob can include up to 50,000 blocks. The maximum size of a block blob is therefore slightly more than 4.75 TB (100 MB X 50,000 blocks). For REST versions before 2016-05-31, the maximum size of a block blob is a little more than 195 GB (4 MB X 50,000 blocks). If you are writing a block blob that is no more than 256 MB (64 MB for requests using REST versions before 2016-05-31) in size, you can upload it in its entirety with a single write operation; see [Put Blob](https://docs.microsoft.com/en-us/rest/api/storageservices/fileservices/put-blob).

I am not able to see the backup in the Azure storage

There is something wrong. It can take time but you should see something.
Do you have anything in logs?

code_blue · January 17, 2018, 3:30pm

The snapshot did get restored after a while. Since we were restoring an existing Index with a different name, the existing index had to be closed during restore we had to manually monitor when the new Index has been restored.

Is there a way of knowing if Index has been restored?

system · February 14, 2018, 3:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.