Hi,
I've exhausted my google-fu and have trawled the documentation and similar issues discussed in these support and am completely stuck.
I'm attempting to snapshot my cluster to GCS, and the system seems to be ignoring my credentials.
elastic version:
"version" : {
"number" : "7.5.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
"build_date" : "2020-01-15T12:11:52.313576Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
built on ubuntu 18.04
GCS plugin version: 7.5.2
This is what I have:
-
A credentials file for my service account in GCS. This is available to every data node, and from each I can use the credentials file along with the GCP CLI and validate that the credentials are correct - i.e. I am able to list, write, and retrieve files into my specific bucket using the credentials. So I know 100% that these credentials are correct.
-
I have then added the credentials file as a secret to elastic search. I have done this on every data node as follows:
/usr/share/elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /path/to/my/credentials.json,
EDIT: I've also run
POST _nodes/reload_secure_settings
against each node to reload the settings. And I've also tried restarting the entire cluster assuming that might force a reload.
- I then pick a node, and execute the call to register the repository, as follows:
PUT _snapshot/my_repository
{
"type": "gcs",
"settings": {
"bucket": "my_bucket",
"base_path": "whatever"
}
}
I get the following error response:
{
"error": {
"root_cause": [
{
"type": "blob_store_exception",
"reason": "Unable to check if bucket [my_bucket_name] exists"
}
],
"type": "repository_exception",
"reason": "[snapshot_archive] cannot create blob store",
"caused_by": {
"type": "blob_store_exception",
"reason": "Unable to check if bucket [Bucket name removed] exists",
"caused_by": {
"type": "security_exception",
"reason": "access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"
}
}
},
"status": 500
}
It seems that the default user's credentials I am supplying are not being picked up. What have I missed?
EDIT: I believe that because I am running within GCE, elastic search is defaulting to the "application default" service account associated with the virtual machine instance, rather than what I've specified in the credentials file.
Many thanks
Guilaume