Monitoring restore progress in Elasticsearch

Elastic Stack Elasticsearch
1

This question deals with getting progress percentage during snapshot restore. I know we can use the RecoveryRequest java API to get totalRecoveredBytes, totalRecoveredFiles, recoveredBytePercent, etc. during the yellow stage of the restore when replicas are being created. Is there any way we can progress percentage during the red stage when the primaries are being created?

2

To a large extent, the restore process is using the same facilities as recovery. So you can use the same mechanism for both processes. See the last two paragraphs of the Monitoring Snapshot And Restore section of our documentation for more detail.

3

Perhaps you are looking for the cat command:

curl -XGET [host]:9200/_cat/recovery | grep -v "done"

That is the simple way to tap your finger while you hurry up and wait.

1 Like
4

I am using Elasticsearch version 2.1.1. I have created a large snapshot that takes approximately an hour to restore. I need to report the progress percentage during the restore so that the GUI can display a progress circle while the restore is occuring. This, of course, will give the user some insight into how long the restore will take.

In order to report the progress percentage I need to compare the total bytes in each shard of each index with the number of recovered bytes for all the indices in the snapshot.

When I issue the following request during the restore:

                curl -s "localhost:9200/_recovery?pretty&human"

I get the "total_in_bytes" and "recovered_in_bytes" values for the indices that have been completely processed or are currently being processed. The "total_in_bytes" field is the total number of bytes for the shard and the "recovered_in_bytes" is consistently increasing until it reaches the "total_in_bytes", as expected. But the problem is the above curl command does not include any of the indices that the restore operation has not yet started processing. If I specifically request an index that I know is in the snapshot, but has not started to be processed:

                curl -s "localhost:9200/hds/_recovery?pretty&human"

I get an empty set "{}".

Ultimately, I need to accomplish this with my software component using the java API. When I execute the following code:

                RecoveryResponse recoveryResponse = client.execute(RecoveryAction.INSTANCE, new RecoveryRequest(currentIndices)).actionGet();

I do not get the indices that the restore operation has not yet started proccessing, like the curl command. But, in addtion, the "totalBytes" field is consistently increasing the same as the "totalRecoveredBytes" field. So each time I issue the above code the percentage complete is 100% when the next time I issue it the two fields are larger for the same exact index.

Any help you can give me to get the restore progress completion would be very helpful

5

Hi Igor, not sure if you will get notified. I responded to your reply this morning

thanks,
Adrienne

6

Yes, I was notified. I just didn't have time to get to it yesterday. Sorry.

You can get this information from the snapshot by running

curl -s "localhost:9200/_snapshot/REPO_NAME/SNAPSHOT_NAME/_status