Snapshot restore fails on Ensuring snapshot repository exists

Joren_Inghelbrecht · May 1, 2020, 10:47am

I want to switch from a CPU optimized deployment (elastic cloud v7.6.1)
to a new IO optimized deployment.

I tried the following methods:

create a new deployment based on a snapshot
create a new deployment and then restore from the current deployment

Both methods fail on the step "Ensuring snapshot repository exists" and I found the following stacktrace in the logs:

[instance-0000000000] Reload failed for plugin [GoogleCloudStoragePlugin] java.io.UncheckedIOException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'service_account': was expecting ('true', 'false' or 'null') 
at [Source: java.io.ByteArrayInputStream@5ba4d05c; line: 1, column: 17] 
at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:203) ~[?:?] 
at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.getClientSettings(GoogleCloudStorageClientSettings.java:163) ~[?:?] 
at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.load(GoogleCloudStorageClientSettings.java:151) ~[?:?] 
at org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.reload(GoogleCloudStoragePlugin.java:79) ~[?:?] 
at org.elasticsearch.action.admin.cluster.node.reload.TransportNodesReloadSecureSettingsAction.lambda$nodeOperation$1(TransportNodesReloadSecureSettingsAction.java:98) [elasticsearch-7.6.2.jar:7.6.2] 
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1621) [?:?] 
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?] 
at org.elasticsearch.action.admin.cluster.node.reload.TransportNodesReloadSecureSettingsAction.nodeOperation(TransportNodesReloadSecureSettingsAction.java:96) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.admin.cluster.node.reload.TransportNodesReloadSecureSettingsAction.nodeOperation(TransportNodesReloadSecureSettingsAction.java:45) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) [x-pack-security-7.6.2.jar:7.6.2] 
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:762) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2] 
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] 
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'service_account': was expecting ('true', 'false' or 'null') 
at [Source: java.io.ByteArrayInputStream@5ba4d05c; line: 1, column: 17] 
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11] 
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11] 
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3528) ~[jackson-core-2.8.11.jar:2.8.11] 
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2686) ~[jackson-core-2.8.11.jar:2.8.11] 
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:878) ~[jackson-core-2.8.11.jar:2.8.11] 
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:772) ~[jackson-core-2.8.11.jar:2.8.11] 
at com.google.api.client.json.jackson2.JacksonParser.nextToken(JacksonParser.java:52) ~[?:?] 
at com.google.api.client.json.JsonParser.startParsing(JsonParser.java:215) ~[?:?] at com.google.api.client.json.JsonParser.parse(JsonParser.java:360) ~[?:?] 
at com.google.api.client.json.JsonParser.parse(JsonParser.java:337) ~[?:?] 
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79) ~[?:?] 
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73) ~[?:?] 
at com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:301) ~[?:?] 
at com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:281) ~[?:?] 
at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.lambda$loadCredential$7(GoogleCloudStorageClientSettings.java:195) ~[?:?] 
at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?] 
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?] 
at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:194) ~[?:?] 
... 50 more

Any idea what the issue might be or if I should report this somewhere?

Luca_Belluccini · May 1, 2020, 10:59am

Snapshots are already automated on Elastic Cloud.
Are you able to list existing snapshots in the Elastic Cloud UI?

If you're not able to list existing snapshots, then open a support case at https://cloud.elastic.co/help

If instead you are able to list the snapshots, then this is what you can do:

Create the new IO optimized deployment
When creating it, select the option Select a deployment to restore from one of its snapshots:

image1239×1154 72.2 KB
You will be able to restore the snapshots sharing the same repository of the old deployment.

Joren_Inghelbrecht · May 1, 2020, 11:09am

I can see the snapshots of my current deployment:

When I create a new deployment, I can select the snapshots:

When I create a new blank deployment, I can select a snapshot:

So that all works as expected, but the execution of the snapshot restore fails with the stacktrace that I mentioned.

Luca_Belluccini · May 1, 2020, 11:24am

Are the 2 deployments in the same region?

If they are, then please open a support request on https://cloud.elastic.co/help

Thank you for your patience.

Joren_Inghelbrecht · May 1, 2020, 11:25am

Yes, the same region. I'll contact support, thank you!

system · May 29, 2020, 11:25am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.