Error when loading Google Cloud Storage credentials file

Elastic Stack Elasticsearch
1

I'm trying to get Google Cloud Storage Repository Plugin going:

I installed repository-gcs plugin:

# /opt/elasticsearch/bin/elasticsearch-plugin install repository-gcs
-> Downloading repository-gcs from elastic
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setFactory
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.URLPermission http://www.googleapis.com/* *:
* java.net.URLPermission https://www.googleapis.com/* *:
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-gcs
# service elasticsearch restart
 * Stopping Elasticsearch Server                                                                 [ OK ] 
 * Starting Elasticsearch Server                                                                 [ OK ] 
#

I then created a keystore and added the string (per Secure Settings).

yet while trying to Create a Repository, I'm getting following error:

# curl --silent --request PUT elk:9200/_snapshot/repository-gcs?pretty --data '{"type":"gcs","settings":{"bucket":"repository-gcs","client":"digaweb"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "Error when loading Google Cloud Storage credentials file"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[repository-gcs] failed to create repository",
    "caused_by" : {
      "type" : "exception",
      "reason" : "Error when loading Google Cloud Storage credentials file",
      "caused_by" : {
        "type" : "i_o_exception",
        "reason" : "The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information."
      }
    }
  },
  "status" : 500
}
#

I'm NOT running elasticsearch inside of Google Compute Engine, so I defined environment variable GOOGLE_APPLICATION_CREDENTIALS , however still getting same error (see above):

# export GOOGLE_APPLICATION_CREDENTIALS=/digaweb-c819f0854eae.json 
# echo $GOOGLE_APPLICATION_CREDENTIALS
/digaweb-c819f0854eae.json
# 
# service elasticsearch restart
 * Stopping Elasticsearch Server                                                                                                                                                                            [ OK ] 
 * Starting Elasticsearch Server                                                                                                                                                                            [ OK ] 
#

Please advise.
Thanks in advance)

2

Which version of elasticsearch are you using? What command are you using to add the credentials file to the keystore?

3

I apologize for not including that information in my topic from the start, but there it is:

elasticsearch:

# curl elk:9200
{
  "name" : "linode5",
  "cluster_name" : "digaweb",
  "cluster_uuid" : "DnyG6zP1QYSrC_kBvZzFZQ",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}
#

keystore's add:

cat digaweb-c819f0854eae.json | /opt/elasticsearch/bin/elasticsearch-keystore add --stdin digaweb

4

You have two problems. First, the add command is for string settings. But GCS requires the entire file be added. Second, you are specifying the setting as digaweb, but that is not the setting name GCS looks for.

See the docs for using a service account file:
https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-gcs-usage.html#repository-gcs-using-service-account

I believe your keystore command should look something like this:

/opt/elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file digaweb-c819f0854eae.json
1 Like
5

per @rjernst comment, I ran elasticsearch-keystore with "add-file" instead of "add --stdin":

# /opt/elasticsearch/bin/elasticsearch-keystore remove digaweb
# /opt/elasticsearch/bin/elasticsearch-keystore list
# /opt/elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file digaweb-c819f0854eae.json
# echo $?
0
# /opt/elasticsearch/bin/elasticsearch-keystore list
gcs.client.default.credentials_file
# curl --silent --request PUT elk:9200/_snapshot/repository-gcs?pretty --data '{"type":"gcs","settings":{"bucket":"repository-gcs"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "Error when loading Google Cloud Storage credentials file"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[repository-gcs] failed to create repository",
    "caused_by" : {
      "type" : "exception",
      "reason" : "Error when loading Google Cloud Storage credentials file",
      "caused_by" : {
        "type" : "i_o_exception",
        "reason" : "The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information."
      }
    }
  },
  "status" : 500
}
#

Please advise)

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

8

It looks like you are adding the setting to the keystore, but not restarting elasticsearch. The keystore is only read on elasticsearch startup.

9

While trying to solve my problem, which resulted in the same error message as above, I stumbled upon this page. This is what I did in order to solve it. When adding the key like so:

bin/elasticsearch-keystore add-file gcs.client.default.credentials_file digaweb-c819f0854eae.json

be sure to set the client in your request, in this case the client is default.

curl -s -XPUT elk:9200/_snapshot/repository-gcs -d '{"type": "gcs", "settings": {"bucket": "repository-gcs", "client": "default"}}'

Source: https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-gcs-usage.html

10

I tried your way and gotten yet another error:

root@elk12:~/_# docker exec -it elasticsearch12 bin/elasticsearch-keystore add-file gcs.client.default.credentials_file 
/digaweb-c819f0854eae.json
root@elk12:~/_# docker exec -it elasticsearch12 bin/elasticsearch-keystore list
gcs.client.default.credentials_file
root@elk12:~/_# curl -u elastic:changeme -s -XPUT localhost:9200/_snapshot/repository-gcs?pretty -d '{"type": "gcs", "settings": {"bucket": "repository-gcs", "client": "default"}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "json_e_o_f_exception",
        "reason" : "Unexpected end-of-input: expected close marker for Object (start marker at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@687b81a1; line: 1, column: 1])\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@687b81a1; line: 1, column: 155]"
      }
    ],
    "type" : "json_e_o_f_exception",
    "reason" : "Unexpected end-of-input: expected close marker for Object (start marker at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@687b81a1; line: 1, column: 1])\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@687b81a1; line: 1, column: 155]"
  },
  "status" : 500
}
root@elk12:~/_#
11

I found a mistake, there was one } missing at the end:

root@elk11:~/elastic/elasticsearch# curl -u elastic:changeme -s -XPUT localhost:9200/_snapshot/repository-gcs?pretty -d '{"type": "gcs", "settings": {"bucket": "repository-gcs", "client": "default"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[repository-gcs] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[repository-gcs] path  is not accessible on master node",
    "caused_by" : {
      "type" : "google_json_response_exception",
      "reason" : "403 Forbidden\n{\n  \"code\" : 403,\n  \"errors\" : [ {\n    \"domain\" : \"global\",\n    \"message\" : \"Insufficient Permission\",\n    \"reason\" : \"insufficientPermissions\"\n  } ],\n  \"message\" : \"Insufficient Permission\"\n}"
    }
  },
  "status" : 500
}
root@elk11:~/elastic/elasticsearch#

I'm running curl command off of master node though...

12

@alexus You need to check your google credentials/permissions. That error means Elasticsearch was not able to write a test file to GCS.

13

I apologize, I actually ran commands in GCE environment and that's why it worked (well, sort of)), however when I run curl command outside of Google Cloud environment, I'm still getting same error as I did before:

# curl --silent --request PUT elk:9200/_snapshot/repository-gcs?pretty -d '{"type": "gcs", "settings": {"bucket": "repository-gcs", "client": "default"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "Error when loading Google Cloud Storage credentials file"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[repository-gcs] failed to create repository",
    "caused_by" : {
      "type" : "exception",
      "reason" : "Error when loading Google Cloud Storage credentials file",
      "caused_by" : {
        "type" : "i_o_exception",
        "reason" : "The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information."
      }
    }
  },
  "status" : 500
}
# docker exec -it elk bash
# echo $GOOGLE_APPLICATION_CREDENTIALS
'/digaweb-c819f0854eae.json'
# /opt/elasticsearch/bin/elasticsearch-keystore list
gcs.client.default.credentials_file
#

Please advise.

1 Like
14

I am getting this too. running v5.6 on EC2 instance