Snapshot restore from 1.7.4 to 2.1.2 cannot find files

Elastic Stack Elasticsearch
1

Hello,
I'm trying to restore an index snapshot created on ES cluster v. 1.7.4 to an ES cluster running version 2.1.2. I was able to register a S3 repository (myRepo), and I can verify that the new cluster can access it.
However when running this command:

curl -XPOST "http://esCluster-2.1.2/_snapshot/myRepo/all_11_28_2016/restore?wait_for_completion=true" -d'
{
"indices": "myIndex",
"ignore_unavailable": true,
"rename_pattern": "(.+)",
"rename_replacement": "index_restored$1"
}'

I get the following error:

"file_not_found_exception","reason":"Blob object [snap-all_11_28_2016.dat] not found: The specified key does not exist.

Which is true, snapshot files stored in S3 are named "snapshot-all_11_28_2016".

Why is ES cluster looking for a different snapshot name? According to this page:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Restoring between the mentioned cluster versions should work.
Is there a way to work around this problem, like by providing the snapshot file name explicitly?
Thanks a lot,

M

2

That should work. Indeed, we switched from snapshot-* to snap-*.dat format in 2.0, but both formats should be supported for read operations. To help us diagnose the issue, could you post here the list of files in the base directory of S3 repository and the output of curl -XPOST "http://esCluster-2.1.2/_snapshot/myRepo/_all?

Related to this, could you also verify that you specified correct base_path and bucket in the repository settings. Was repository verification successful, when you registered it? When you try to restore and get the error back, do you see the corresponding error message in the log file on the master node? If you do, please post the entire error message with stack trace here.

3

Hello,
The command you provided:

curl -XPOST "http://esCluster-2.1.2/_snapshot/myRepo/_all"

Returns this message:

{"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[myRepo:_all] Invalid snapshot name [all], must not start with ''"}],"type":"invalid_snapshot_name_exception","reason":"[myRepo:_all] Invalid snapshot name [all], must not start with ''"},"status":400}

I didn't have "base_path" field in the repository creation body, only "location" field. I modified the command to use base_path, then tried to restore the snapshot, but restore operation failed with the same error ("Blob object [snap-all_11_28_2016.dat] not found: The specified key does not exist.").
In both cases repository creation succeeded, I got this response from curl:

curl -XPUT "http://esCluster-2.1.2/_snapshot/myRepo" -d '{
"type": "s3",
"settings": {
"access_key": "***",
"secret_key": "***",
"bucket": "myBucket",
"region": "us-west-2",
"base-path": "mySnapshots", <<< this was "location"
"readonly": true
}
}'

{"acknowledged":true}

In S3 myBucket/mySnapshots folder contains the following:

index (file)
indices (folder)
metadata-all_MM_dd_yyyy (files)
snapshot-all_MM_dd_yyyy (files)

I'm going to talk to the ops team to get the master node log output.
Thanks a lot,

M

4

Hello,
I pasted log entries from master node log file below. I only copied the messages that seemed relevant to repository/restore issues.
Thanks a lot,

M

Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshot(BlobStoreRepository.java:445)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:177)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:82)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:41)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:185)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.openInput(S3BlobContainer.java:109)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.common.blobstore.support.AbstractLegacyBlobContainer.readBlob(AbstractLegacyBlobContainer.java:62)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:93)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:83)
Nov 30 17:40:12 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshot(BlobStoreRepository.java:439)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshot(BlobStoreRepository.java:445)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:177)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:82)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:41)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:185)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.openInput(S3BlobContainer.java:109)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.common.blobstore.support.AbstractLegacyBlobContainer.readBlob(AbstractLegacyBlobContainer.java:62)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:93)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:83)
Nov 30 17:40:54 es-master-3.novalocal docker[1538]: at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshot(BlobStoreRepository.java:439)

5

Sorry, wrong command, I meant curl -XGET "http://esCluster-2.1.2/_snapshot/myRepo/_all" (GET instead of POST)

6

Hello,
I executed the command you provided, it returned this message:

{"snapshots":[]}

Does that mean anything?
Thanks a lot,

M

7

It means that elasticsearch doesn't see any snapshots in the path that you specified. Unless you are using an entire bucket, you need to specify correct base_path and it had to be spelled correctly (i.e. with underscore base_path and not with dash base-path).

8

Hello,
It looks like that was the problem. Changing field name to "base_path" (underscore) helped. After running the GET command you provided I was able to see a lot of snapshot entries.
I was also able to start the restore operation.
Thank you for your help!

M

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.