Can't create snapshot using GCS plugin


(diego wentz antunes) #1

Dear Group,

I'm trying to create a snapshot for my elasticsearch cluster into Google Cloud Storage - GCS using
the repository-gcs plugin but I'm finding difficult to debug or find the correct solution.
I'm currently running the EFK stack on GKE.

Here is what I already did.

First, I'm using elasticsearch 6.3.2 as you can see here.
{
"name": "ewbuIJE",
"cluster_name": "cluster-test",
"cluster_uuid": "dle1raa-Q-uauokuBMYDpg",
"version": {
"number": "6.3.2",
"build_flavor": "default",
"build_type": "deb",
"build_hash": "053779d",
"build_date": "2018-07-20T05:20:23.451332Z",
"build_snapshot": false,
"lucene_version": "7.3.1",
"minimum_wire_compatibility_version": "5.6.0",
"minimum_index_compatibility_version": "5.0.0"
},
"tagline": "You Know, for Search"
}

I created another elasticsearch docker image based on 6.3.2 with the repository-gcs plugin already installed.

I followed the documentation from the following links as they point to my current version.
Snapshot Modules
repository-gcs

Created a bucket using:

gsutil mb -p [PROJECT_NAME] -c [STORAGE_CLASS] -l [BUCKET_LOCATION] gs://[BUCKET_NAME]/

To add my service account to the keystore I used the following commands:

cat > sa.json << EOF
<SERVICE ACCOUNT CONTENTS>
EOF

Then:

bin/elasticsearch-keystore add-file gcs.client.default.credentials_file ./sa.json

So when I list the contents of the keystore with the command bin/elasticsearch-keystore list
it shows that the key is there.

  • I run the commands to add the service account into the keystore on all the nodes.

Then I restarted all the nodes using:

service elasticsearch restart

This is the command I used inside Kibana:

PUT _snapshot/test-bucket
{
  "type": "gcs",
  "settings": {
    "bucket": "test-bucket"
  }
}

This is the output:
{
"error": {
"root_cause": [
{
"type": "repository_verification_exception",
"reason": "[test-bucket] path is not accessible on master node"
}
],
"type": "repository_verification_exception",
"reason": "[test-bucket] path is not accessible on master node",
"caused_by": {
"type": "google_json_response_exception",
"reason": "403 Forbidden\n{\n "code" : 403,\n "errors" : [ {\n "domain" : "global",\n "message" : "Insufficient Permission",\n "reason" : "insufficientPermissions"\n } ],\n "message" : "Insufficient Permission"\n}"
}
},
"status": 500
}

And this is the log from elasticsearch:

[2019-03-14T14:28:01,674][INFO ][o.e.r.RepositoriesService] [ewbuIJE] update repository [test-bucket]
[2019-03-14T14:28:01,769][WARN ][r.suppressed             ] path: /_snapshot/test-bucket, params: {repository=test-bucket}
org.elasticsearch.repositories.RepositoryVerificationException: [test-bucket] path  is not accessible on master node
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:582) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:210) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:413) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:398) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:64) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(MasterService.java:515) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService$AckCountDownListener.onNodeAck(MasterService.java:613) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService$DelegetingAckListener.onNodeAck(MasterService.java:555) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery$1.onNewClusterStateProcessed(ZenDiscovery.java:362) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.discovery.zen.PendingClusterStatesQueue.markAsProcessed(PendingClusterStatesQueue.java:177) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery$3.clusterStateProcessed(ZenDiscovery.java:796) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.ClusterApplierService$SafeClusterStateTaskListener.clusterStateProcessed(ClusterApplierService.java:537) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:489) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:626) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.3.2.jar:6.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Insufficient Permission",
    "reason" : "insufficientPermissions"
  } ],
  "message" : "Insufficient Permission"
}

Looks like there is some issues with my service account permission, I reviewed and gave it reader and writer permission.

Can someone provide some tips or clues as I'm probably missing something.

Best regards, Diego