[s3-repository] path is not accessible on master node

Elastic Stack Elasticsearch
1

Hello Team,

We are trying to configure the snapshot of indices and store over S3 bucket. But we are facing issue when we are trying to configure repository.

We are using 3 node cluster and we instlled the repository-s3 plugin on all the 3 nodes as well as we configured the S3 access in elasticsaerch keystore also on all the 3 nodes.

We can see below that we have repository-s3 plugin on all the 3 nodes:

root@elk-1:/usr/share/elasticsearch/bin# curl -XGET -u elastic:xyz  "http://x.x.x.x:9200/_cat/plugins?v&s=component&h=component,version,description&pretty"
component     version description
repository-s3 7.10.2  The S3 repository plugin adds S3 repositories
repository-s3 7.10.2  The S3 repository plugin adds S3 repositories
repository-s3 7.10.2  The S3 repository plugin adds S3 repositories

We have restarted the elasticsaerch service over all the 3 nodes.

But when we are trying to configure the S3 repository using the below command from Dev Tools in Kibana

 PUT _snapshot/s3-repository
{
  "type": "s3",
  "settings": {
    "bucket": "our-bucket-name",
    "region": "eu-central-1"
  }
}

We are getting the below error:

 "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3-repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3-repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-1HFytCKGQQytGjCTrctjMA/master.dat] using a single upload",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "amazon_s3_exception: /YYYYMMDD/REGION/SERVICE/aws4_request\". (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 70BD7993AABE7A1D; S3 Extended Request ID: 9KmfSE3OIdqzSWjo6UWYRZeY5gv2qmkytXIkmw2EO7874eJHoM4vlUXPnu2qFy32+nct+wREaSg=)"
      }
    }
  },
  "status" : 500
}

We have checked the multiple questions which are already asked by other users in the forum but nothing is worked for us.

Can you please help us on this issue?

Any help or guidance will be appriciate.

Thank You

2

This is a weird error, but a quick web search suggests it might be because your bucket name is already in use in a different account/region.

Try using a bucket with a different name, and/or check that the bucket you're trying to use does actually exist in the right account and region.

3

@DavidTurner, Thank You for your quick response.

Yes, this bucket is exist in our account.

We were using same bucket earlier also when we were using ELK version 6.4. Now we are not using older servers and we deleted all the older machines and created new machines and installed ELK 7.10.2.

But we want to use the older bucket to store the snapshot.

We deleted the older Access Key and Secret Key for that IAM user and created new one and we are using them now.

is it feasible to use the same bucket and configure the repository or we need to create new bucket?

Thank You

4

@DavidTurner , We have created new bucket and tried with that bucket but still we are getting same error.

Thank You

5

I don't know then, sorry, I've never seen this error before. Elasticsearch is using the official Amazon SDK to talk to S3 so I don't understand what else might be causing a malformed header. Maybe double-check that your access key and secret key are in the keystore correctly, with no extra punctuation or spaces or anything.

1 Like
6

@DavidTurner, Thank You for your response and help.

Finally able to fix this issue. I have tried many things but nothing worked for me. Then I remove the repository-s3 plugin from all the 3 nodes and install again and configure the access key and secret key again in elasticsaerch-keystore and then restart the elasticsaerch service over all the 3 nodes.

This may be cause off the issue.

Thank You once again for your helping hands.

Thanks

2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.