Elastic snapshot for GCS seem to be ignoring my client credentials

Guillaume_Roderick · February 2, 2020, 1:00pm

Hi,

I've exhausted my google-fu and have trawled the documentation and similar issues discussed in these support and am completely stuck.

I'm attempting to snapshot my cluster to GCS, and the system seems to be ignoring my credentials.

elastic version:

"version" : {
    "number" : "7.5.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
    "build_date" : "2020-01-15T12:11:52.313576Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"

built on ubuntu 18.04

GCS plugin version: 7.5.2

This is what I have:

A credentials file for my service account in GCS. This is available to every data node, and from each I can use the credentials file along with the GCP CLI and validate that the credentials are correct - i.e. I am able to list, write, and retrieve files into my specific bucket using the credentials. So I know 100% that these credentials are correct.
I have then added the credentials file as a secret to elastic search. I have done this on every data node as follows:

/usr/share/elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /path/to/my/credentials.json,

EDIT: I've also run

POST _nodes/reload_secure_settings

against each node to reload the settings. And I've also tried restarting the entire cluster assuming that might force a reload.

I then pick a node, and execute the call to register the repository, as follows:

PUT _snapshot/my_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
   "base_path": "whatever"
  }
}

I get the following error response:

{
  "error": {
    "root_cause": [
      {
        "type": "blob_store_exception",
        "reason": "Unable to check if bucket [my_bucket_name] exists"
      }
    ],
    "type": "repository_exception",
    "reason": "[snapshot_archive] cannot create blob store",
    "caused_by": {
      "type": "blob_store_exception",
      "reason": "Unable to check if bucket [Bucket name removed] exists",
      "caused_by": {
        "type": "security_exception",
        "reason": "access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"
      }
    }
  },
  "status": 500
}

It seems that the default user's credentials I am supplying are not being picked up. What have I missed?

EDIT: I believe that because I am running within GCE, elastic search is defaulting to the "application default" service account associated with the virtual machine instance, rather than what I've specified in the credentials file.

Many thanks

Guilaume

Armin_Braun · February 2, 2020, 3:28pm

Hi @Guillaume_Roderick

this rather looks like an issue with the Java Security Manager than with your credentials from that error message alone. If the below suggestion does not help, could you share the full error message for this that you can find in your logs?

Before that though: Could you please restart the nodes after adding the credentials to the keystore instead of just reloading secure settings. The GCS plugin's secure settings don't support hot reloading so that's probably why the GCE environment credentials would be picked up.

Guillaume_Roderick · February 2, 2020, 3:50pm

Hi Armin,

Yes I tried restarting the nodes, and no joy.

Here's the full error message:

==> /mnt/extended/elasticsearch/messages_cluster.log <==
[2020-02-02T15:49:00,547][WARN ][r.suppressed             ] [MY BUCKET NAME] path: /_snapshot/snapshot_archive, params: {repository=snapshot_archive}
org.elasticsearch.transport.RemoteTransportException: [clients-dev-prod-elastic-master-node-g1tf-master][10.0.0.13:9300][cluster:admin/repository/put]
Caused by: org.elasticsearch.repositories.RepositoryException: [snapshot_archive] cannot create blob store
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:338) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:841) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:240) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.2.jar:7.5.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Unable to check if bucket [clients-dev-prod-ally-elastic-snapshot-archive] exists
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:118) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.<init>(GoogleCloudStorageBlobStore.java:89) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:94) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:42) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:334) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:841) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:240) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.2.jar:7.5.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) ~[?:?]
Caused by: java.lang.SecurityException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
	at java.lang.Class.checkMemberAccess(Class.java:2848) ~[?:?]
	at java.lang.Class.getDeclaredFields(Class.java:2247) ~[?:?]
	at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:175) ~[?:?]
	at com.google.api.client.util.ClassInfo.of(ClassInfo.java:90) ~[?:?]
	at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:198) ~[?:?]
	at com.google.api.client.util.ClassInfo.of(ClassInfo.java:90) ~[?:?]
	at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:198) ~[?:?]
	at com.google.api.client.util.ClassInfo.of(ClassInfo.java:90) ~[?:?]
	at com.google.api.client.util.GenericData.<init>(GenericData.java:74) ~[?:?]
	at com.google.api.client.util.GenericData.<init>(GenericData.java:55) ~[?:?]
	at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:146) ~[?:?]
	at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:129) ~[?:?]
	at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:102) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:450) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:429) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:336) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:313) ~[?:?]
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:264) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:82) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31) ~[?:?]
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:77) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:153) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:118) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.lambda$refreshAndClearCache$0(GoogleCloudStorageService.java:67) ~[?:?]
	at org.elasticsearch.common.util.LazyInitializable.maybeCompute(LazyInitializable.java:103) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.LazyInitializable.getOrCompute(LazyInitializable.java:81) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:92) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:95) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:115) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:115) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.<init>(GoogleCloudStorageBlobStore.java:89) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:94) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:42) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:334) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:841) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:240) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.2.jar:7.5.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.2.jar:7.5.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) ~[?:?]

Armin_Braun · February 2, 2020, 4:11pm

@Guillaume_Roderick

this does indeed look a lot like it's not picking up the right credentials/project_id. Could you try manually setting the correct project_id as a repository setting as documented here to see if that helps?

Guillaume_Roderick · February 2, 2020, 5:32pm

Including the project id now gives the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "storage_exception",
        "reason": "storage_exception: Anonymous caller does not have storage.buckets.get access to clients-dev-prod-ally-elastic-snapshot-archive."
      }
    ],
    "type": "repository_exception",
    "reason": "[snapshot_archive] cannot create blob store",
    "caused_by": {
      "type": "blob_store_exception",
      "reason": "Unable to check if bucket [my bucket name] exists",
      "caused_by": {
        "type": "storage_exception",
        "reason": "storage_exception: Anonymous caller does not have storage.buckets.get access to clients-dev-prod-ally-elastic-snapshot-archive.",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "401 Unauthorized\n{\n  \"code\" : 401,\n  \"errors\" : [ {\n    \"domain\" : \"global\",\n    \"location\" : \"Authorization\",\n    \"locationType\" : \"header\",\n    \"message\" : \"Anonymous caller does not have storage.buckets.get access to clients-dev-prod-ally-elastic-snapshot-archive.\",\n    \"reason\" : \"required\"\n  } ],\n  \"message\" : \"Anonymous caller does not have storage.buckets.get access to [my bucket name].\"\n}"
        }
      }
    }
  },
  "status": 500
}

Armin_Braun · February 2, 2020, 5:40pm

@Guillaume_Roderick

one question. You're writing you've added the credentials to every data node. Do the master node(s) also have those credentials set up in the keystore? If not they should.

Guillaume_Roderick · February 2, 2020, 7:32pm

Hi,

Yes I've added to the masters also.

Is there a way of debugging both what's in the keystore, and whether it can be retrieved? I.e. is there a call I can make to elastic that would print out the default client's settings?

Guillaume

Armin_Braun · February 2, 2020, 9:15pm

@Guillaume_Roderick

I'm afraid not and that's on purpose. You wouldn't want to be able to accidentally log a secure setting like this in clear text.

I think your best option in making sure that this isn't an issue with the keystore would be to try and add the same settings under a different client name (i.e. replace the default in your keystore add common with some other name) and then use that client name when creating the repo instead of default. That would make it clear whether or not the ES nodes see the setting since they would complain about the client not existing if there's something up with the keystore.

Guillaume_Roderick · February 3, 2020, 8:55am

Hi,

Yes - I've actually already tried that (I didn't want to send you two problems at once), and have a separate (but I believe related) issue:

I added my key file as follows (to every node)

/usr/share/elasticsearch/bin/elasticsearch-keystore add-file gcs.client.alternate.credentials_file /path/to/key/file.json

and restarted each node

and then sent the following request:

PUT _snapshot/my_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "base_path": "whatever"
    "client": "alternate"
  }
}

I get the following error

{
  "error": {
    "root_cause": [
      {
        "type": "blob_store_exception",
        "reason": "Unable to check if bucket [my_bucket] exists"
      }
    ],
    "type": "repository_exception",
    "reason": "[snapshot_archive] cannot create blob store",
    "caused_by": {
      "type": "blob_store_exception",
      "reason": "Unable to check if bucket [my_bucket] exists",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Unknown client name [alternate]. Existing client configs: default"
      }
    }
  },
  "status": 500
}

According to the docs, I need to add client settings to my elasticsearch.yml file (see: https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs-client.html) ... now the confusing thing here that is that the docs indicates that all values have defaults, or are inferred from the credentials... so there would seem to be nothing additional nothing additional to add once the credentials file is in the keystore.

I think this is related to the other problem as I suspect that they key store is just not being loaded.

Guillaume

Armin_Braun · February 3, 2020, 9:35am

Yes this is the issue. The secure setting alone in the keystore is enough of a setting to make ES create the named client, no need to add anything to the elasticsearch.yml when it comes to the GCS plugin.

Could it be that you are in fact running elasticsearch from a different path than the path of your keystore accidentally?

Also, just as a sanity check, could you check that your new settings actually show up in:

/usr/share/elasticsearch/bin/elasticsearch-keystore list ?

Guillaume_Roderick · February 3, 2020, 10:26am

Hi Armin

Elastic is running as follows:

elastic+ 24676 6.5 63.0 4342004 2384096 ? Ssl 10:15 0:31 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=COMPAT -Dfile.encoding=UTF-8 -Dio.netty.noKeySetOptimization=true -Dio.netty.noUnsafe=true -Dio.netty.recycler.maxCapacityPerThread=0 -Djava.awt.headless=true -Djna.nosys=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+AlwaysPreTouch -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseConcMarkSweepGC -XX:-OmitStackTraceInFastThrow -XX:CMSInitiatingOccupancyFraction=75 -Xloggc:/var/log/elasticsearch/master/gc.log -Xms2g -Xmx2g -Xss1m -server -XX:MaxDirectMemorySize=1073741824 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch/master -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch-master.pid --quiet

so from /usr/share/elasticsearch/

I'm then using the keystore at: /usr/share/elasticsearch/bin/elasticsearch-keystore

and list gives me (on all nodes)

> /usr/share/elasticsearch/bin/elasticsearch-keystore list

gcs.client.default.credentials_file
keystore.seed

Guillaume

Armin_Braun · February 3, 2020, 10:40am

The paths look correct as far as I can see. Did you make sure to run the commands to add the setting and create the keystore using the same user that is running the elasticsearch process itself?

Guillaume_Roderick · February 3, 2020, 11:34am

Yep, elastic search is running as user "elasticsearch", and I also added the credentials as that user.

Guillaume_Roderick · February 3, 2020, 12:13pm

I'm going to head down this rabbit hole now:

"reason": "access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")"

Armin_Braun · February 3, 2020, 1:06pm

I think investigating the exception won't be all that helpful to be honest. It looks like we failed to set the appropriate security manager exclusions to allow the GCE credential fallback to run (and make its API requests to figure out the current project_id etc.). This is just a symptom of the keystore contents not showing up for ES itself.

Could you check whether all files and subdirectories under /etc/elasticsearch are properly owned by the ES user?

Guillaume_Roderick · February 3, 2020, 1:46pm

Yes - everything is owned by the ES user.

Armin_Braun · February 3, 2020, 3:12pm

I think I may have found the underlying issue here.

Could you try adding the the setting of ES_PATH_CONF=/etc/elasticsearch in /etc/sysconfig/elasticsearch suggested there?

Guillaume_Roderick · February 3, 2020, 4:47pm

Wow!

Ok, so I've fixed it but it wasn't quite as you suspected:

I had already set ES_PATH_CONF in my environment, but, I'd set it to /etc/elasticsearch/config which is where I'm writing all my configs (jvm options etc).

Elastic Search was working fine with this setup and honouring the config path.

The gcs repository plugin was actually ignoring ES_PATH_CONF, and explicitly expecting to find all configs in /etc/elasticsearch irrespective of what ES_PATH_CONF was set to.

So by moving ES_PATH_CONF to /etc/elasticsearch it fixes the issue in that elastic is able to run with the configs from the new location, and then the gcs repository works.

Thank you so much for all your help! This has been a journey

system · March 2, 2020, 4:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.