Snapshot from GCS to AWS Cluster (repository_missing_exception)

Elastic Stack Elasticsearch
1

Hi All,

We are migrating our ES cluster from GCP to AWS. Now the repository we were using for snapshot and restore is of type gcs. My new ES cluster on AWS is running. And I want to restore the data that was snapshot by GCS cluster last night to AWS cluster index.
If I run the restore API with same name as GCS repository name. It is giving error as repository_missing_exception.

Also on restarting the ES cluster on AWS servers. Some error related to bucket exists is coming. Please find the error below :

[2019-01-23T13:17:48,480][WARN ][o.e.r.RepositoriesService] [node1] failed to create repository [gcs][gcs_repository]
org.elasticsearch.common.blobstore.BlobStoreException: Unable to check if bucket [wallbl-es5-backup-m] exists
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:118) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.(GoogleCloudStorageRepository.java:119) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.lambda$getRepositories$1(GoogleCloudStoragePlugin.java:129) ~[?:?]
at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:304) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:495) ~[elasticsearch-6.0.0.jar:6.0.0]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_111]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:492) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:479) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:158) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-6.0.0.jar:6.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_scope",
"error_description" : "Empty or missing scope not allowed."
}

Please help in solving this error and restoring the data that was snapshot by GCS cluster last night to AWS cluster index.

Thanks in advance.

2

I'm wondering why you are using a so old version? And not latest 6.5.x?

How did you create the repo on the new cluster? What exact commands did you run?

3

I'm wondering why you are using a so old version? And not latest 6.5.x?

Well it was a tech lead call not to upgrade till it is not really neccassary.

How did you create the repo on the new cluster? What exact commands did you run?

Well the cluster was cloned from GCS cluster to AWS cluster by other agency. Now we need to setup the cluster with all the data that was snapshot in GCS cluster and not updated in new cluster due to time taken by the cloning process. So i have not created any repo on new cluster.

4

:roll_eyes:

I'd not do that but recreate the repository on the new cluster.
Drop the repository and create it again IMHO

5

I'd not do that but recreate the repository on the new cluster.
Drop the repository and create it again IMHO

And how to restore data from the snapshot taken by GCS cluster on new cluster.

6

With the restore API once you have defined the repository again.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.