Getting repository verification exception

Elastic Stack Elasticsearch
1

I am trying to create snapshots of my ealsticsearch in s3 bucket. i had created an s3 bucket with public access and created an user with the following permission

       {
"Statement": [
    {
        "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:ListBucketMultipartUploads",
            "s3:ListBucketVersions"
        ],
        "Effect": "Allow",
        "Resource": [
            "arn:aws:s3:::elastic-snapshots"
        ]
    },
    {
        "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:AbortMultipartUpload",
            "s3:ListMultipartUploadParts"
        ],
        "Effect": "Allow",
        "Resource": [
            "arn:aws:s3:::elastic-snapshots",
            "arn:aws:s3:::elastic-snapshots/*"
        ]
    }
],
"Version": "2012-10-17"

}

after that installed s3 repository plugin by using this

sudo bin/elasticsearch-plugin install repository-s3

and added secret-key and access key by following commands

   bin/elasticsearch-keystore add s3.client.default.access_key
   bin/elasticsearch-keystore add s3.client.default.secret_key

and tried to create snapshot_repository with following command

curl  -X PUT "10.0.0.198:9200/_snapshot/snapshot_repository?pretty" -H 'Content-Type: application/json' -d'{ "type": "s3", "settings": { "region": "us-west-1",
"bucket": "elastic-snapshots", "base_path": "/snapshot-repository/",
"endpoint": "s3-us-west-1.amazon.aws.com" } }'

it is giving me the following exception

     "error" : {
"root_cause" : [
  {
    "type" : "repository_verification_exception",
    "reason" : "[snapshot_repository] path [/snapshot-repository/] is not accessible on master node"
  }
],
"type" : "repository_verification_exception",
"reason" : "[snapshot_repository] path [/snapshot-repository/] is not accessible on master node",
"caused_by" : {
  "type" : "i_o_exception",
  "reason" : "Unable to upload object [/snapshot-repository//tests-72t6ssc0TC2oQXObyby9yQ/master.dat] using a single upload",
  "caused_by" : {
    "type" : "sdk_client_exception",
    "reason" : "sdk_client_exception: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"
  }
}
},
     "status" : 500
    }

any help would be appreciated. i'm literally struck here.

2

Hi @hello5346

It looks like the default client settings are not getting picked up and that's why the logic is falling back to the EC2 metadata credential resolver which is failing here.

After adding the default credentials to the keystore, did you restart your nodes or reload the secure settings via the API? I would expect either a restart or calling that API to fix things and allow you to add+verify the repository.

3

Hello @Armin_Braun, Thanks for the answer here is what i had done and my result

I had restarted the system and started elasticsearch and ran this

    curl  -X PUT "10.0.0.198:9200/_snapshot/s3_repository?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-snapshot"
  }
}
'

and the output is still same

 {
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3_repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-orwYhYx0ShiMyd6EFC7DIQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"
      }
    }
  },
  "status" : 500
}

expecting a reply from you.

4

@hello5346
could you share a few more details on your setup please:

What version of Elasticsearch are you using?
Did you make sure to run

   bin/elasticsearch-keystore add s3.client.default.access_key
   bin/elasticsearch-keystore add s3.client.default.secret_key

on all of your cluster nodes?

5

@ Armin_Braun
i'm using 7.6 elasticsearch with 3 node cluster with xpack enabled

 bin/elasticsearch-keystore add s3.client.default.access_key
 bin/elasticsearch-keystore add s3.client.default.secret_key

I ran the above commands and provided both the keys and after that i had restarted the machine and services as well. I can manually upload the data folder of elasticsearch into the s3 bucket using aws-cli but i want this snapshot-repository to be in place.

I tried this out again after providing the keys:

    curl --user elastic:elastic  -X PUT "10.0.0.198:9200/_snapshot/snapshot_repository?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "region": "us-west-1",
    "bucket": "elastic-snapshot",
    "endpoint": "s3-us-west-1.amazon.aws.com"
  }
}
'

{
  "error" : {
"root_cause" : [
  {
    "type" : "repository_verification_exception",
    "reason" : "[snapshot_repository] path  is not accessible on master node"
  }
],
"type" : "repository_verification_exception",
"reason" : "[snapshot_repository] path  is not accessible on master node",
"caused_by" : {
  "type" : "i_o_exception",
  "reason" : "Unable to upload object [tests-WNnM6viCRb-sgs-YYfBPfQ/master.dat] using a single upload",
  "caused_by" : {
    "type" : "sdk_client_exception",
    "reason" : "sdk_client_exception: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"
  }
}
  },
  "status" : 500
}

can you identify anymore steps i'm missing.

6

Thanks @Armin_Braun
This has worked for me.

added s3-repository plugin in all the nodes by using this

bin/elasticsearch-plugin install repository-s3

after that provided the access-key and secret_key

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

restarted the machine's and services and after that created the snapshot with the following command

curl --user elastic:elastic -X PUT "10.0.0.6:9200/_snapshot/my_s3_repository?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-snapshot"
  }
}
'
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.