GCS repository backup issue

Elastic Stack Elasticsearch
1

Hi

We have been using GCS repository for backing up our easlticsearch data from 7 months. But from last 1 week, we are getting 503 Service Unavailable error while creating a snapshot.

Error Stack Trace

[gcs_repository_wall:snapshot_2018_10_29/fqOMR7njQumykOwTdHqZ5Q] failed to create snapshot
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:335) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.snapshots.SnapshotsService.beginSnapshot(SnapshotsService.java:364) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.snapshots.SnapshotsService.access$700(SnapshotsService.java:105) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.snapshots.SnapshotsService$1.lambda$clusterStateProcessed$1(SnapshotsService.java:282) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0.jar:6.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
{
  "code" : 503,
  "errors" : [ {
"domain" : "global",
"message" : "Backend Error",
"reason" : "backendError"
  } ],
  "message" : "Backend Error"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145) ~[?:?]
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) ~[?:?]
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) ~[?:?]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432) ~[?:?]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)

Our ES is 6.0.0. Anyone, please suggest something. This is a very critical issue as it occurred 4 times in the last 8 days.

2

It sounds like an error on google storage side.

Probably not related but I'd suggest anyway to upgrade your elasticsearch version to a more recent and stable version than 6.0.0.

3

thanks for the reply @dadoonet.

I am checking this issue from google storage side also. Please share if you know anything related to this issue.

We didn't plan yet to upgrade elasticsearch. But most probably will start in January next year.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.