Elasticsearch Keystore Authentication Failure

I have a GCP bucket, and machines running with ES 7.6.1 in two separate clusters:

  • cluster 0, with machines "machine 0"
  • cluster 1, with machines "machine 1"
  • GCP Bucket

My goal is:

  1. register a repository with this GCP Bucket on machine 0
  2. snapshot to this bucket with machine 0
  3. register a repository with this GCP Bucket on machine 1
  4. restore from the snapshot created by machine 0

For both machines, I have:

  1. installed repository-gcs
  2. added the credentials to ES: elasticsearch-keystore add-file gcs.client.default.credentials_file <service_account.json>
  3. POST localhost:9200/_nodes/reload_secure_settings to reset the secure settings.
  4. Restart ES (only necessary on older versions of ES, but I tried it anyway)

From there I am able to:

  1. Register a repository with this GCP Bucket on machine 0
  2. Snapshot to the bucket
  3. Restore from this snapshot back to machine 0

However, I am not able to register, snapshot, or restore following the exact same steps on machine 1. I have tried at least 10 times on multiple different machines from each cluster and am able to reliably interact only with machines from cluster 0.

For machines from cluster 1, I get this authentication error:

curl -X PUT "localhost:9200/_snapshot/test-backup?pretty" -H 'Content-Type: application/json' -d'
> {
>   "type": "gcs",
>   "settings": {
>     "bucket": "<bucket>",
>     "base_path": "backup"
>   }
> }
> '
  "error" : {
    "root_cause" : [
        "type" : "blob_store_exception",
        "reason" : "Unable to check if bucket exists"
    "type" : "repository_exception",
    "reason" : "[test-backup] cannot create blob store",
    "caused_by" : {
      "type" : "blob_store_exception",
      "reason" : "Unable to check if bucket exists",
      "caused_by" : {
        "type" : "security_exception",
        "reason" : "access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"
  "status" : 500

On machines from cluster 1, I can even access the bucket using gsutil, so I don't believe it's a network configuration issue.

I am completely out of ideas as to why two machines, with the same version of ES, configured the same way with elasticsearcy-keystore with the same service account would behave differently in this scenario.

Any suggestions would be greatly appreciated.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.