HDSF backup restore no progress info

Elastic Stack Elasticsearch
1

Hi,

I have a problem with restoring backup through hdfs plugin (elasticsearch/elasticsearch-repository-hdfs/2.1.0-hadoop2)
on elasticsearch cluster version 1.6.0

I was restoring 12GB data (20000000 documents) backup from hdfs.

After calling:
POST _snapshot/CLUSTER_NAME/SNAPSHOT_NAME/_restore

I got answer ok, but

  • GET _snapshot/_status - does not return any info about restoring

  • GET _cat/indices - shows added restored index in state red with only one element per shard.

  • no info in server logs

  • second try to run restore result in error message f.e.
    { "error": "ConcurrentSnapshotExecutionException[[CLUSTER_NAME:SNAPSHOT_NAME] Restore process is already running in this cluster]", "status": 503}

  • GET _cat/shards
    index_name 0 p INITIALIZING 8.55.24.233 node1
    index_name 0 r UNASSIGNED
    index_name 1 p INITIALIZING 8.55.24.234 node2
    index_name 1 r UNASSIGNED

  • the only log that restore is taking place is 15 IOPS and 4MB per sec on disk write

  • and finally after 30 min data stared to grow in the index. and ended in 7 minut.

Why does no progress info is presented?

2

Anything in your ES logs that might be of use?

3

Even after changing logs into debug level there was no single line about restore.

4

To close the thread @costin has helped me in the issue on the github:

      To quote the docs:

The cluster health operation provides only a high level status of the
restore process. It’s possible to get more detailed insight into the
current state of the recovery process by using indices recovery and cat
recovery APIs.

In other words, see

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-recovery.html and

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html

That is for recovery, use the dedicated endpoint APIs.